Here are the steps to enter the desert:
You will be given 'Dust Covers' by talking to the Mercenary Commander, after you have completed the jungle.
You must take them to the wizard at the Inn, and he will enchant them for you and allow you to equip them.
You must return to your room, enter the 'Armoury', and equip the 'Dust Covers' from the crystal.
You can return to the 'valley', the area with the purple sludge and Shambling Mirages. Proceed to the top of the area, slide down the rope, and walk to the north to enter the desert.
Do NOT go through the hole in the wall. The entrance to the desert is the right-side.