Some messing around with the physics seems to have fixed it. Smooth as butter now, but collisions shoot you off in the opposite direction.
//giphy.com/embed/d2ZbK5RWbGuqYBe8
