Thanks for the feedback and glad you're enjoying it! The situations you described seem to be where the direct route from the origin to the destination is blocked by a wall/roof/fence. This is one of the limitations of the game engine's pathfinding system (or quirk is a nice euphemism!) - it does not route around walls (or similar items), so it does require the player to find a route around the obstacle.
SPOILERS AHEAD
If you've reached the cave, you've also reached the ruins. I found the pathfinding quirk useful here because the player has to figure out the path to a semi-hidden treasure there, and can't just click on it to go there directly. :)