Drink milk and eat beans to increase your XP; this way you'll be stronger in duels.
Use a knife to increase your bad reputation; this way the bandit will leave you alone.
Don't get too bad a reputation, or the cops will get you.