Yes this is very tricky to deal with during development. I'd say, get a lot of people to playtest it (which you probably have), so you have a good range of people to adjust the difficulty accordingly. Also, I realised that in a lot of situations, I did not mind the difficulty if there is something that shows me I am close to reaching the goal and the game gives me feedback. In your case, you can see cracks forming in the rock, which shows me how damaged it gets. If I did not have that, or if the visuals and audio were not here to keep me engaged, I may not have been so patient. So again, well done!