One day I will update this part, I don’t like how it works as well (because of imperfect scripting). But at this moment you can try to set the easy mode of mini-games in the options -> gameplay options, it will make things a little bit easier.
In any case, I will think what I can do with this.