From my experience, you have to slow down when changing rooms, because if you trigger a fight and walk into another room during the "trigger fight" animation it'll bug and do what you've encountered. By pausing in front of rooms I was able to complete it :)