What about how the ball is always reflected in a vertical angle when hit by either end of the bat?
Yup, noticed, I change the collision shape to be a capsule instead. That would help, but not completely.