Could you make it so walls do not affect the ground layer?  Or at least as a toggle option?  Adding and removing  walls does odd things to the ground, usually things I don't want it to do.