I did give gems unique negative effect, which is designed to be insignificant when played once, but accumulate quickly when playing multiple and/or “stacked” gems.
Still not 100% sure if that is sufficient downside, but at least it should make player hesitate a bit when acquiring/playing gems.