Thanks for taking the time to give so much feedback! All your ideas are pretty cool!
I may create a rogue lite with the concept and I'll surely experiment with your ideas a little.
I am against hotswapping though. To swap a specific item you either have to get rid of all items on the left or all items on the right. This can lead to some tough choices. (I didn't implement enough item variety and enemies for this choice to be meaningful in any way though)