For some reason, I was well above the correct position when in VR, but sitting on the ground fixed the problem.
Great take on the theme, lots of fun and could become a nice game it you feel like polishing it. ;)