Thanks for playing!
I did consider your gem removal and swapping idea, but decided I wanted the decisions with gems to be a bit more permanent. In the game concept, the gemstones would "physically" alter the turrets structure (thus irreversible), but I didn't get the turret sprites finished, so it's less apparent.
I do intend to add a turret selling mechanic that refunds most of the cost of the turret and gemstones.