I think I'll have to go with using WASD to scroll. Right now, they are linked to the gamepad movement in a weird way but that's my problem.
