Sometimes barriers and doors that are removed don't update the pathfinding correctly. I don't know why--been hunting that bug for a long time and tried a few things without success. Saving and reloading has always cleared it as that forces the map to refresh.
You will need followers to progress through the Horny Dragon's Hoard quest for both loyalty and fertility shrines.