This is not a good excuse. If the territory belongs to other fraction, then it has every right to forbid the dwarves from passing through their territory.
The whole territory belongs to the villagers and the owners of the castle. Goblins and bandits parasitize on these territories. Remember the lord of the rings.
