Good news, I've found out what causes the bug. If you time out during the arrow QTE, it'll desync and throw an error on any future ones.
For now, while I debug it. If you want a smooth playthrough, you'll have to start a new save and not time out during the QTE part. However, you can completely fail it and it'll still proceed fine, just don't time out.
Thanks for finding the bug!