I've got controller issues too.
It seems like during gameplay it reads the "Up" and "Down" directions on my joystick a little wrong and makes the character veer to the right or left when I try to go straight up or down. I don't have this problem in menus or when I use the keyboard.
(I'm using an 8-Bit-Do Neo-Geo style pad BTW.)