You have to click the "Equips" armor piece next to "Map", then click the slot of the part of clothing you want to unequip.
