Ah dang, still happened. Made sure that option was unticked. This time I was in Grasslands of Ort, moving northeast I think. I've also had it happen when moving from the secret room in the Chroniclary back to the main room.
I've recently updated the game to 1.3.1, which includes a change that might solve the issue you are having. Please give it a try!
