The party members you recruit during the time skip phase are removed when you leave the time skip phase, and they join your party again when you go back into the time skip phase.
Whether they are currently in your party or not doesn't matter. If you had the event to receuit them then you have that requirement and are missing something else.