The problem isn't really that stuff getting destroyed is annoying. Imagine a big banner that says "in 5 minutes, you will level up and everything will be destroyed".
The problem is that those 5 minutes are now boring, there's nothing to do in them. (If you reset $ too, I might spend it but the minutes are still pretty boring)