There is one encounter on the Castle Town side. It and the Merchant both have the same "pre-encounter" text, though.
Swift Shield is a strong choice for Damon. You could also increase your defense with Earth, smack him with a few Power Busters to weaken him and make his Dark Barrier useless, or even try the Catalytic Base formulas to blast him down with one hit using the Catalytic Trigger. There's also some gear and recipes you can unlock that can trivialize the fight if you leave him for later.