Just go to the bandit camp and get back to the map. Now if you hover camp icon you should see their morale. It's dropping every day, unless you do some quests or bribing. If morale is lower than 25 then the bandits starts attacking again. I think sparring should work too, as it counts as a normal fight.