Sorry, I think I wasn't clear about the drifting to the right. It was not the movement that's drifting to the right, but rather the camera/view is slowly turning to the right. Also I don't have a gamepad on me, so I don't think that's the problem.
I tried the game again and it's still happening. Though if I press the ESC button, it suddenly stops drifting. I can still walk and look around. It only starts drifting when I focus on the game again by clicking the screen.
After further testing, it seems like the issue only occurs when I don't enter fullscreen mode.