Oh wow, thank you so much for taking the time on creating a screencast and reproducing the bug!!!
I can reproduce it every time. It seems to depend on wether you have selected a gem before entering the menu and if a swap would theoretically be possible in the direction of the d-pad pressed.
Knowing what? In one condition is just a case (selection active or not) missing. Fast fix. Additionally any active selection is now released before entering the menu.