You can find them somewhere in Helgem, and also can use the dictionary once you've found their weaknesses.
I think the dueling to raise affinity is unbalanced I dueled Rafaelo 6 times I hardly made any progress. At this point I need to duel like 50+ times for several characters
