As you progress through the story, you can summon them from the portal room and they will be in the castle afterwards.
