Yes you can talk them only in the places they are at. For plot reasons, many stays in Grimoire (Vaeril, Galan, Dasyra) but the others need to stay in their own places. So that's the correct behavior :)
