You're not crazy- this is a bug which I passed over twice before patching. The pin in the final door is supposed to match the key fragment with a hole in it. Sorry about that!
