I now played it with some friends and now I see that moving up and down does make a difference!
I wish there was some feedback for what I did wrong when I missed the ball, I lost everytime must be the games fault!
Just kidding of course, really enjoyed the game :)