I think I figured out that the light areas (which shrink when you lose) give you a speed advantage when you've hit the ball, and the dark areas give the advantage to the Racket of Doom... But if you can't make the ball go to that side of the court besides when you serve it, then it seems to be mostly up to the RNG...