Removing materials with explosives makes absolute and total sense.
Try making the destructed objects either not have a collision or apply physics to them so that if you run into them they move, as it is they can block you from getting into rooms and you have to blow them out of the way again.