I think only siege units should be able to destroy walls. And melee units should be able to climb ladders so they do not destroy a whole wall to kill a archer. It is really anoying when you make a army to attack a castle and all of your units die because they try to kill units on walls. Melee units should be able to destroy gates and focus on them more.