Probably the problem is that there is no direct fight with slavers in the game. You can trick them, you can run away from them, and you can also sneak attack them. And the result always depends on chance and skill level. But for some reason there is no direct fight - only with the guards in the gold mine.
This applies to both the slavers in the desert and the slavers who took over the milk farm.