In the next update, I'll add ladders to the rooms after the sidequest completes, so hopefully that might make it a bit easier.
