Yeah, that makes sense. I got the agent working on something along those lines this morning... I'll post back if it succeeds.
The main reason I have it doing an "infinite world" is just that I am not an artist so wanted pure procgen producing the game world. But I should be able to use that to pre-bake a sizable hunk of the world, I would hope, and do less on-the-fly generation at runtime. Then, as you say, fall back to JIT loading if the player exceeds the large pre-baked initial area.