Since the map has edges, I think it would be cool to spawn them from there. You could have raiders as well as trade caravans. For traders, you could require a special trade outpost building only available once you've made contact. They could be like the Harrapan trade settlements in Sumer.
As for the raiders, you could step up the raids as the map becomes cleared of barbarian tribes, or some other metric of game progression. Maybe they could even try to form their own colonies in addition to capturing yours.