I have a question: how exactly do you manage to move to a specific location while you are talking to people? Sometimes it works when I'm in a conversation with someone, and sometimes it doesn't. For example, I talk to Darian and say let's go to Rosalyn together, and boom, we're in the store. Then we talk about heading back to the Watchtower. Although the text describes us returning and so on, we actually remain inside Rosalyn's shop. At first, the conversation implies that we are at the Tower, but later on, text cues appear that suggest we are still inside Rosalyn's shop.
You, the player, don't. It's up to the NPC to decide and they're very imperfect. Though you can suggest or hint at it to encourage them to, it's not guaranteed. Try stretching out the RP of you going to... wherever, so the AI has more chances of just taking you there. Make up an excuse that you forgot something and have to go back to pull the scene back to your actual location to start the attempt over. That's what I'd do. If it doesn't work after the second try, I'd wrap up the conversation there, since they aren't playing along, why bother to continue. Try again another day.
You can simply correct it in an explicit manner while out of character for the AI to do such, format is like this; '(OOC: Player should've been moved to X location based on the conversation, please move us there for the sake of consistency)' And the AI will usually comply without breaking character.
I'm working on improving this for Gemma 4 specifically using its currently unused thinking mode. Models with low active parameter count struggle with this task the way it's currently set up. Qwen 3.5 seems to do fine. I have not extensively tested it with the dense Gemma 4 or DeepSeek, but if it's a problem with those too, please let me know.