I prefer a region-based system rather than a distance-based one. I feel like it would be more realistic, as it would enforce natural borders like rivers and mountains. Maybe have a mix of both, with a small influence radius that doesn't expand or not very much, and otherwise go by region. So if you settle near the border between biomes you can go a bit on the other side, but you're otherwise bound to natural borders. Something I feel is important.
If the borders of settlements and maskling territories follow rivers and mountains, rather than being circles that ignore terrain, then border friction feels more natural. If the masklings attack you because you built a tower two tiles too far to the left and its circle overlapped with their invisible circle, it feels arbitrary and annoying. If you piss them off because you crossed a clearly visible, unchanging boundary, then it makes them feel more human. "This is our side of the river, this is yours" is much better than "You can get within 144 squares of our huts, if you get even one step closer we'll shank you."
Expansion should in my opinion bound to milestones allowing you to claim more regions as "core". Those milestones could simply be population (IMO something exponential like 150-350-600-1000 rather than a flatter 200-400-600-800), wealth, or later in development be tied to achievements like successfully developing a noble class or technology.
>I think most or all of my cities would fail that check.
Looking at the cities you posted, at a glance Scalestander port is on a single region, it passes. Grasschopper port looks to be on four regions, but at 1600+ citizens I think it could be allowed one or two more, so it passes. Wallstander on the other hand seems to sit on three regions, and at 229 it should be allowed two, so if the region-based system was in place, you most likely would've had to stay on one side of the lake, maybe making your town more vertical instead.
>I would also limit how closely settlements can build to each other, requiring them to merge administratively first.
Forbid settling into another settlement area of influence (core regions + regions adjacent to core), can't expand into another settlement's area of influence unless it's for a merge.
>I'm unsure of the best way to effectively limit settlement size.
I think you'd need more systems in place, like happiness or diseases, before an adequate soft-cap can really be implemented. As it stands, hauling inefficiencies and FPS death act as a soft cap.