This is an excellent concept! This can actually be a really good VR game.
One suggestion I had was to maybe have the mouse move the camera without "pausing" the console.
Had a minor camera issue in which the reticle wasn't at the center, so picking up objects was a little awkward sometimes.
Great work! :)