I did have my controller plugged in, but I was playing with keyboard + mouse. I think if I was experiencing stick drift my character would be trying to walk on his own, right? I'll have to do some more testing later.
Played a bit more after I got home from work. A lot of the problems I had earlier seems to have gone away. Maybe something just screwed up on the first boot?