Find the swordsman in tavern to get the key to the cliffside temple.
(I'm assuming you've spoken to the hermit already.)
