Happy to help.
I probably should've looked at the Info/FAQ Sheet, that's on me. Sorry.
I like the idea of this being "Maximum Effort" difficulty rather than some random number. We all know a bad run on dice rolls can make or break even the most balanced quest, so anything you have that's consistent is going to work.
That being said, I still don't like your formulas and I'll try to come up with my own, maybe some kind of percentile that ramps up. Like the 2nd enemy increases all monsters by 10%, the 3rd by 25%, the 4th by 45%, and so on.
Re - Custom Monsters: I would encourage you to have some heroes fight a 611 monster (Chainsaw Goblin) and a 161 monster (Dread Warrior with a Dagger. I would happily define them as being in the same difficulty ballpark, whereas you have the Goblin twice as difficult as the Dread Warrior. The 611 is unlikely to survive the first hero attack(s) and therefore be unable to use it's 6AD attack, where the 161 monster is going to stand there and be hammered on while chipping away at the Heroes health on its turn... potentially. Maybe the heroes roll poorly when attacking 611 and 161 can't roll a Black Shield to save it's life... literally... see what I did there? Again, subjective but I would argue the 161 would be far more difficult for brand new heroes than 611 (assuming the heroes get a chance to attack first). It would be fun to try out.
No way, 8 Goblins would chew up 1 Abomination. He'd take 1, maybe 2 out and would himself be taken down in the 2nd or 3rd round.
Maybe that's the way to set a difficulty. Goblin is the "base" difficulty, and every other enemy is rated on how many goblins it takes to kill it (best 2 out of 3). We got all winter to figure this out, then it's WoM time and no one will care about Homebrew for a while! :)