Someone gave you the directions already, but yeah.
The system doesn’t work like a map, it’s just a combination of directions in a specific order like a password.
And it’s set up so if you go to a “null” direction, the game will make you stumble upon a location you haven’t seen before.
Locations you have seen you can check on the Remember option. Also, the locations in the remember thing are kept through saves.
But yeah, the Goblin Witch, you find her then she tells you to go all the way south (S->S->S) after that when you find her hat you’re supposed to check it in your inventory to know where to find her next.
Burning building is in the town, it even gives you hints if you check other places.
But yeah, you’re not the first one to dislike the explore system.