If you mean the bandit camp, then do the story quest where you cure the shopkeeper from his sickness and visit him the day after giving the cure to him, then visit that town on day 20 or beyond to trigger the Bandit attack story event and you should be able to get there following the scenes from it.