Boy am I late to this comment. I overcame this issue by altering the controls for the game. Just set the grip touches to grip buttons and delete the actual grip buttons. Done.
My new problem is that the swords are orientated for the Oculus Touch, so they're a pain to swing.