The doors in the Dragon dungeon seem to be bugged and opening them still will not let the player through them.
(Edit, I don't know how but I somehow fixed it. I think going into a battle or leaving the cave fixes it?)
(Second Edit, I have 3 other companions and I still can't complete the loyalty shrine.)
Viewing post in [18+] The Restoration of Aphrodisia comments
Sometimes barriers and doors that are removed don't update the pathfinding correctly. I don't know why--been hunting that bug for a long time and tried a few things without success. Saving and reloading has always cleared it as that forces the map to refresh.
You will need followers to progress through the Horny Dragon's Hoard quest for both loyalty and fertility shrines.