Another little glitch and I think I know what makes it happen. When you remove a character from your party, they are also removed from their Soul Room, if they are in one. I think. I will check this myself if you don't check it first.
Before removal with same character housed:
After removal with same character housed:
However, you are experiencing the disappearance of housed souls?