Hi Balian,
Thanks for the feedback. Sounds like you've been affected by some of the bugs that I've been having difficulty tracking down.
Also are you using an Oculus Rift or a Vive? The Rift is currently not compatible (although I haven't specifically turned it off), but I've been getting reports of a couple users with hard to hit buttons using the Vive too.
That said, I've made a number of improvements to the climbing system based off feedback from others; hopefully once I rerelease/update, people like yourself will find it much more intuitive!