I have a similar problem to you, I refuse to play without the mod, I can't stand seeing NPCs in worlds where they are not from, but it is true that the mod is causing a lot of lag during gameplay, even on the loading screen when traveling to any lot, I also did the same tests as you and the mod is the cause.
I think we should wait for a future update of the mod.