You mentioned "going straight back keeps hitting me into buildings I can't see when I go backwards", which leads me to think maybe your mouse look broke somehow? Because to go back you can simply look back and go forward like any 3rd person adventure game.
