I think the dragon cave might be bugged? I just wandered the caves for a irl hour and have not encountered the gems for the potions and there are dragon soldier patrols like if I sided with the dragon on a save where I fought the dragon
Viewing post in [18+] The Restoration of Aphrodisia comments
The gems are intentionally hard to find as the dragon potions are meant to be hard to craft. Dragons have few penalties that matter much by that point in the game where the party will likely have multiple tens of thousands of gold.
Do you the player or a companion invested in Foraging to increase the odds?