You can afk inside the ball, making the ball push or collide with the player might prevent that, or just disabling the ball's hitbox while it is in contact with the player
or the ball's hitbox could be disabled while it isn't moving
Yea I figured that out right off the 'bat' too. Good suggestions, making it always move and not slow down all the way might help too.
