Thanks!
Controller sensitivity is a thing. It's different for different people. Some people have the problem you do, some don't.
So I've decided that the best solution is to add a controller sensitivity option. Added onto the to-do list! Thanks for the feedback, it's helping. :)