So after testing again:
I'm on Windows 10 x64
Using Firefox 54 x64 the camera jumps every 1.5 - 2 scds, I captured a video of it: www.sylvain-maurice.com/misc/timmy_camera_jumps.flv
Using Chrome 59 x64 the problem disappears completely.
So at this point I don't think it comes from the camera but rather that it's due to performance consumption spikes in firefox.