What do your ATL transforms look like?
To be honest, I'm not sure this is a good use case for the controller viewport, or even a viewport at all. Given the animations you want to do, I'd probably just manually assign left/right to swap whatever card is in the center, and use ATL animations to move them to the right location. The cards wouldn't even need to be buttons; you can just have a variable track which one is selected and perform an action accordingly when the button_select action is performed.