Here is the settlement info
At around 500 to 550 population the people skip't frames and the game became slower (basically lag).
I have a pretty good pc so i think for people that have lower specs they are going to begin lagging at around 300 People.
Don't know about the building lag but more pixels on screen = more lag so it will still lag when you have a lot of People at the cost of them being homeless because 600 people = 600 houses = 600x the lag
Btw i learned how you can control warbands and migrants now. Apparently you have to click the next button under you map which is really great so far maybe i'm going to start another run.
If you want anymore info i'll gladly show it.