Ah, yes the game is quite hard for people that aren't used to rhythm games.
To effectively swim you need to alternate left and right click in sync with the beat. It heavily punishes mashing buttons, while the argument part on the left is much more forgiving on the timing.