It is indeed the physics material. The solution I found was to make a second collider slightly larger than the first above the characters feet and give that material no friction. That way your feet aren't butter but your sides are. There are probably better solutions out there though.