When you reach the desert entrance and the game informs you that you cannot pass, you're teleported back to the Quiet Meadow campsite. That should be your first clue your new goal isn't back there... we're not going to bring you back just to make you return in that direction.
At that point the game also should pop up with a little message saying you want to go track down the adventurer trio, and that they left for the south. Considering the areas you visited were the "northern bridge camp" and "northern settlement" and "northern desert"... it should hopefully click that you want to go the opposite direction from all that.