They're found as foraging nodes, so it's best to go around with a the player or companion invested heavily in Foraging. You can brew a potion to give you a temporary Foraging +2 bonus as well.
Viewing post in [18+] The Restoration of Aphrodisia comments
The gems are intentionally hard to find as the dragon potions are meant to be hard to craft. Dragons have few penalties that matter much by that point in the game where the party will likely have multiple tens of thousands of gold.
Do you the player or a companion invested in Foraging to increase the odds?