Okay, I think not finding the key was probably my fault. I played a few more rounds and managed to find it every time.
However, when the key spawns very low down, almost on the floor, it's basically impossible to pick up. I think it's simply too far below the player's head/camera.