Instead it could look for other (smaller) walls etc. which can be found in other parts of the map.
The wall has to be big enough to fit all the soldiers that are retreating. Sometimes this wall happens to be at edge of scene, but not always.
I could send some soldiers to occupy a building near the defense wall. That would make it a bit more difficult.