If it matters in reproducing, I clicked the Prestige button multiple times because it looked like it wasn't responding at first. That said, when I select continue now, the buildings are back, so I guess that's a way to restore them!
