Try to use a nodraw material for faces that can't be seen, and grouping the walls together.
All the walls on the map or just sections of the walls?
Join the sections that have weird shadows where they intersect, basically
