Were you playing on a game pad? It could explain drifting while standing still (didn't have time to implement dead zone adjustment)
Edit: Replied before reading part you said you played on pad xd. I am using cinemachine and those bugs ate caused by moving player after map is built (in this version i didn't implement one major optimalization to map generation yet)