Yeah, this is an issue that has existed since the games inception. I kind of wish the dev would disable NPC's going into follow mode on their own, and instead just make it sort of work like the "gives you permission to sleep here for the night" thing. Where when it triggers that they'll follow you, the game will just remember that, and then you have to go into the menu and select to ask them to follow you instead.