Sometimes it's easy to miss a small detail in the rules, especially when the creator uses shared tables. 😅
Sounds like a good delve! You cannot leave with the treasure or clues as the only way to the Downtime Phase is through victory or defeat. You would treat leaving the delve as Defeat:
"Defeated players remove all treasure from their Treasure Tracker and proceed to the Downtime phase." (p. 28)