That's just not the way the combat system works - middle and high attacks both hit upper armor, and if an enemy's chest is bare (either because they had no upper armor to begin with or it was destroyed), the attack isn't affected by any armor, even if they're wearing leg armor. You don't have to destroy both. And enemy attacks are just as subject to armor as character's are.