I'm trying to recreate this bug, but have been unable to do so. Did you use the Party Switch menu option from an old save file from a previous version? Or were you playing the story and the party change was forced as part of the plot? If you're using an older save and updated the game, it's possibly a compatibility issue. You can always upload and link your save file prior to the change, and I can try to duplicate the bug with it on the current version.
There is a mechanic that updates player levels so that heroes not in the party don't fall behind in experience gains. The calculations and implementation only occur on party switch. It's also dependent on specific variables that have been tracking total hero levels the entire game (recall you can rebirth a hero to restart at level 1 with current stats intact). If that data is missing, the mechanic could malfunction.