That's my biggest issue with that minigame too. It barely rewards you for doing well and then punishes you heavily for not doing so which sucks when the window for a correct key press can be so small.
It feels like youre not meant to win, which is just bad game design