There's a setting for NPC Action System Accuracy.
Try turning that up when you want something specific.
Then you can turn it back after you've got it.
Other than that, try to be as descriptive of the garment as possible.
Weigh down the word cloud made by the tokens by including words describing the type of the fabric, the style of it, any kind of tailor argot you can throw in there.