I totally understand your idea, the reason it is working like this right now is due to high development cost of such feature for single building type. I have full time job and this feature, unfortunately, would consume significant amount of time to implement and test, while impact on overall gameplay would be negligible (especially with progression). So, this is why in my opinion this problem have low priority right now and I’m trying to “hide” it by reusing already existing and working mechanics. But I will think about further boost of the default builder’s speed (or making it property of the city hall to be increased with progression).
I'm not asking you to fix it, and fix it urgently :) My job is to find the bug and tell the developer about it. And let him decide for himself what and how to do with it :) Logic usually has to be connected only to explain why it needs to be done. Developers are different :) I've been talking to one recently. He created an allegedly incremental game. And there are no incremental mechanics in it, which is what I told him. So he told me that I was an idiot and that I misunderstood the meaning of incrementality. When nothing increases anywhere, these are also incremental games :) Let's say warcraft, starcraft and other rts :)
So do what you want and at a pace that is convenient for you. I just explained why this moment looks illogical to me. Although it practically does not affect the gameplay. Just great games usually differ from just good ones in that they pay attention to the little things.
One of main reason I’ve pushed for public release is to get important feedback on early stages. On the other hands, this is why I had to sacrifice not only small details, but big ones too. Maybe after I finish proper population logic it will be possible to use unassigned citizens as builders with some penalty.