So I obtained all four queen gems, but when I try to slot the final one for the door, I get the "It seems a gem can be placed here" message. I think one of the chests didn't actually add one of them to my inventory.
That's weird. Are you sure you found all 4?
I think it was just a one-off glitch. When I reloaded from an earlier save, and went through again, it let me slot them all.