Continue east, following the castle and then you can exit through its basement. Follow the path past the ruins until you reach the cave.
