Thanks for the constructive criticism; it was an experiment as all game jam entries should be ;-)
For my next game I'm going to keep this mechanism but also offer a non sliding option which should give the best of both worlds. Or maybe just allow people to skip the sliding, I don't know. Having now put in traditional straightforward swapping in that game I am finding myself gravitating to that most of the time. So I think even as the mother (or rather, father) of this game I see the potential flaw in this design too.