There is story significance later on. But if the minigame frustrates you, you can always turn it off for a more classical VN choice experience.
