just a suggestion, perhaps you could implement an exit for conversations. it might not fix the underline issue but it'd be a nice temporary patch at the very least.
its definitely a game that looks like its got plenty of potential though.
from what i gather at least it kind of has a skyrim/fallout vibe with a slight hint of that adult content magic splashed here and there like a nice spice.