But… strangest thing… just reloaded my save - picked random teams and it worked and now I can switch parties…
I also saw it switch Vahn’s sprite with the selected character when the first teammate I picked was the lead of my active party going in…
so now my best guess is that it’s something to do with state of the active party going into the location combined with how you select your team…