Thanks for the feedback!
I think your suggestion is really good, making it more of a side mechanic. You definitely would feel more attachment and weight to your decisions, having built or more directly managed the village yourself. Glad you enjoyed it!