I've got an idea - maybe limit the max (wandering) Normies and Cultists to say, 20 or so. (Or maybe 100, whatever number you think is a good one)
Maybe have an extra building to increase your Normies cap - say, 2x as much as church capacities? And you build those in conjunction with churches. Can call those Houses or Shelters or something.
Show both buildings (Church / House or Shelter) on the map, with the windows slowly lighting up as they near capacity. So in canon, those extra Cultists / Normies you don't see are inside those buildings, and you greatly reduce lag.
(Alternatively, the option isn't to "build" but upgrade your Church / Shelter so it's the same building we see at the start, and it makes sense there's only one of those)
And with the lag reduced there's also the possibility of adding anything extra (demons or magic or what have you) without also killing your screen rate, if you have any such ideas for those in future.
Edit: if there's excess beyond the capacity it just doesn't show up, but yeah the gist is you cap the number of wandering units in the map by shoving the rest into those buildings.