Hey, I refuse to play without this mod as well, but I have noticed that I have been getting unending loading screens when loading into a new lot with this mod. I tried reinstalling the game, taking everything out but this and townie demographics that is how much I love this mod, (I refuse to play without it) but it seems it this is the culprit. Has anyone else experienced this?
I have a similar problem to you, I refuse to play without the mod, I can't stand seeing NPCs in worlds where they are not from, but it is true that the mod is causing a lot of lag during gameplay, even on the loading screen when traveling to any lot, I also did the same tests as you and the mod is the cause.
I think we should wait for a future update of the mod.