I had this same drift issue at first, but I was able to fix it by unplugging my controller and plugging it back in while still in the game. Maybe this will fix it for you also!
Dev note: not sure how to fix this exactly, but it definitely has something to do with a controller being hooked up when game starts and/or controller/keyboard interference at beginning?