Unfortunately, that seems to be a bug some are running into. Are you using the most recent version (1.4)? It should be a bit more stable there.
Otherwise, you can interact with the wall directly south of the blue pedestal in the Hall of Orbs to "return" party members that are lost.
Thank you for playing!