well i think it helps you to known as a "bitch". But if u keep being passive and won't fight with them, you cannot get items which are drop from them. And about bandits, they will take your money for every lose. I don't think you need a reputation around them for now. it doesn't matter to the way the game is now.