Why can't we buy land, build a mansion, upgrade it with things like a pool and dungeon, train members of our personal harem and have a place to store gear and treasure? Nice though it is, I don't think we should permanently live above Vivian's. Anyone else agree?
Really the issue is that affection isn't actually tracked. Affection is instead per-girl, and is arbitrarily set to a new value when a waypoint is reached. For a few characters, this works well enough. For every character, it would require a new system or manually adding those waypoints to each character's events.
As for the mansion, it's just not feasible AFAIK. A new area requires a lot of coding, and something so (relatively) minor would take up a lot of time that could instead be spent expanding the already-extant game. Your best bet is to check the Discord and see if any modders are up for the challenge.