Cool concept!
I would suggest starting the player off with an ability. Some player agency is needed to make the starting pong matches fun.
I'd also suggest making the velocity and direction of the paddle affect the ball's trajectory on bounce. Right now there is very little the player can do to affect the outcome of the starting matches with only their movement.
With these two additions alone the player will see more of the fun and cool content you've added (upgrades, map, bosses). Add in some "juice" to the pong portion and I can see it being super fun!