Thanks for playing! The rarity gives a bit more stats, but we couldn't make it too significant because there's no way to swap them out later. So the base rarity items are already pretty strong. Personally I really liked the rarities an an excuse for giving the gems different colors :)
Ideally higher rarities would be obtainable through a separate mechanic and would give new stats as well, but there's only so much you can do in 3 days!