The issue is that some of the clues are too vague.
For example once of the rooms says "green was never one of our colors", and there is a gray, blue, or green door. However, green appears gray when you activate the orb, and the blue door doesn't lead to progress.
The final clue is something like "you are not one of us" and every door leads to the orb room.