I think the game might have a slight issue with registering your button press (or maybe I'm just too slow) either slightly delayed (after the arrow has already moved out of the target area), or maybe it registers the "de-press" instead. Point is: I could swear I sometimes pressed the button right when the arrow was in the target area and the fishing scene resolved itself with the arrow having already moved outside of the target area.
In case there actually is an issue with the game registering the button press too late, you should aim to press the button as soon as the arrow enters the target area, maybe even try pressing the button shortly before that, see if it helps.
But if you can't do it, don't worry: There are only one or two situations where the game requires you to catch a fish and outside of that, you can just gather mushrooms instead, you won't miss out on anything by ignoring the fishing option.