when I attacked the castle and it went empty, when i went to attack other towers they racked up in population really quickly, as well as my total population.