Since it seems you haven't gotten an answer yet, I'll explain (if you're still wondering). What happens is that when you're in the air you always pull yourself towards the block. But if you are on the ground (or on a ladder) you always pull the block towards you. It's made to be like that and it leads to some interesting puzzles later in the game.
P.S. It doesn't matter if you press or hold wasd