Was it an unfinished house? Selecting any unfinished structures will prioritize builders you have selected to build that structure but won't reassign anyone. The outline has to be green for reassigning.
I really appreciate the feedback though and I'm glad to hear you like the game.