Hmm, yeah. Different people are always going to read at different speeds, so dialogue timing will always be a challenge.
Have you considered having some kind of different presentation for dialogue from offscreen/out-of-scene characters? That might help communicate that being able to move away is intentional and provide players with cues on where they need to go if they want to go back to where the speaking character is.