You're right, you can drop down there and hold on to the ledge. You can do it by jumping off the edge while facing backwards, or by running towards the edge and turning around right before falling. It's pretty tough but you're almost past the hard part!