Finding the pyramid is really confusing me, maybe I am doing something wrong, or I am just bad at understanding the desert maps.
You follow the maps you buy in town the same way you followed the first map that led you to town. Although note that since the maps lead to different places, they obviously can't all lead to the pyramid.
(If you're farther in area progression than that and need more hints, then ask for them more explicitly! I'm just assuming you haven't even found the rewards from following the maps yet.)