Unique idea and pretty challenging at first. There is some jitter when moving around and I think it's because of moving the character in Fixed Update while the camera moves and rotates in Update, so if the physics run at 50 fps (defualt in Unity) and the game runs at like 120 fps then it's really visible. It's kinda tricky to fix that in Unity, but you can just use Cinemachine even for first person game, and that fixes it.