I might be wrong, but I think the villages are bugged. Most of the time I don't get the options of to trade, to occupy, or to destroy. I enter a village and then Rowan watches the attack from a hill.
In my current playthrough, a fully built castle, minus the arena, cannot be supported by 1 ocuppied village and 1 trading village without me adding to the treasury.