Asking villagers to follow you in town was unintentional. I'll do by best to make it up to you by adding town sex content this month.
Can you add few random NPC to town? For lore they will be portal travelers that came to rest and went to adventure soon. Also it'll give lore opportunity to make some buildings for trespassing: tavern or bar (possible sidequest), brothel (why not ;) ?)...
By the way, what about unlockable NPC? During travel you might find a character that need your help and then it appear in town (You might give these NPC role of some altars but with additional money requirement. This will make dye and patterns more useful since you won't need to carry them with hope to find altar)