Unfortunately this is just a case where you have a bad enemy matchup for your build. Fire isn't necessarily a big deal, but you weren't killing the summoned monsters fast enough so they just started multiplying and stacking that burn damage on you.
These fights where something is summoning other monsters basically boils down to: Can you eliminate the summons quickly enough to give you a turn to deal damage to the summoner? If not, you're in for an uphill battle.