I have not been able to reproduce this issue with an OEM Xbox controller on Ubuntu. (Note that only Ubuntu is technically supported, if that makes a difference.)
Do you have another controller you can try? Does your joystick have any drifting issues? (The sensitivity setting can help with this as it adjusts the deadzone.) Please provide more info if you are still experiencing the issue.