We are working on a VR project where you can cast spells in Harry Potter style. You can play the prototype here: https://ravingbots.itch.io/vr-magic-gestures-ai
You can also find there the source code of the project, which would be a nice starter kit for recognising 3D gestures in your VR game.