It was mainly before I got the green gem - it's very easy to make your way over to the yellow area towards the left of start with just the red gem, fall down the pit there, and then be unable to get back up as you can't climb infinitely without a second gem.
I think I also got softlocked by getting somehow boxed in on a flag somewhere in that region? I don't remember the details of how that happened though