You need to bring a key after the first boss!
I think mine glitched then, as the stairs after the key took me to the caves.
