Look for a section of the cliff that's symmetrical. It should be marked with a sparkle.
Viewing post in How do I get into the castle Kharos?
https://imgur.com/XbAQG1a
The door should appear on interacting with the wall in that spot. Don't mind the path color, I took a pic from a later point in the game.