Since I had the same feeling about losing clothes and colors changing, I'll share what the reason was for me:
In the wardrobe, you can have two outfits for each category. The icon is split in two (left and right) and depending on where you click it will select one outfit or the other. By default, most secondary outfits are empty (naked). The icon is not very clear IMO and it's even less noticeable when playing on a phone.
After you perform an action, Sam may automatically switch to an appropriate type of outfit and can pick either variation.
So maybe that's what happened to you: Sam changed to a variation of the outfit that was empty or a different color than what you expected.
By the way, the underwear and stocking icons are also split in two so I kept thinking that I was losing my panties or bra depending on where I was clicking until I understood how it worked.