I think it helps knowing that you can use your keypad arrows. We’ll definitely look into fixing this particular mini-game as it creates the most problems.
I have the same problem and tried to play it with the Keypad arrows. It changed the required pattern every time, I was one away from finishing the sequence.