I figured it out, but for anyone else that gets stuck here like me:
When you get outside, hit the "back" button in the bottom right.
It's not obvious, and it would be appreciated if maybe in this one instance it could be a map icon or have "Town" floating over it.