It was the strangest thing on my first playthrough I was completely stuck inside that library room and couldn't get out, but on my second playthrough I was able to leave the library whether I had the key or not.
I was able to get a lot farther on my second playthrough and had fun with how trippy the map is, had fun with it!