After a few more games, it is obvious that the paddle is very unfair. There should be only a small section on both sides of the paddle that bounces back the ball in the same direction it came from. Currently, it's split in half. So the ball is coming from the right side and hitting the right side of the paddle, always bouncing back the ball in the right again. And for the next landing, we need to move the paddle all the way across.
So either paddle speed should be faster to get the ball to land on the side that we want, even if we are extreme opposite side of the screen, or just make the paddle bounce logic as the middle part of the paddle just inverts the Y direction and at both sides of the paddle the small area change both X and Y.