The gems are intentionally hard to find as the dragon potions are meant to be hard to craft. Dragons have few penalties that matter much by that point in the game where the party will likely have multiple tens of thousands of gold.
Do you the player or a companion invested in Foraging to increase the odds?