On my second playthrough I realised I could spam the arrow key during the pendulum to succeed. In other words, there's no 'cooldown' or penalty for incorrect timing. Given I was getting stuck for a distractingly long time otherwise, I was kinda relieved!
I did notice that the timing for when the hit registered wasn't consistent (or at least, when the number popped up). Like it could just be starting to touch the centre line, or just leaving it, or even beyond it.