I agree that there are several mechanics that are poorly explained. However, some things are probably due to something you missed. Things are easily missed, of course. A log would be pretty cool.
Was there only a single enemy? Did you have the "unstable", "grappled", "netted"(?) or "discord" status effect? Unstable is caused by terrain, grappled and discord are abilities that some enemies have. Also, when an enemy that knocks you back into a wall, you get a "crunch". I believe each of these multiply damage by 1.5.
Another posibility is that the enemy gained damage somehow. They can get the berserk status, and some enemies have other ways of increasing their damage: Orc Berserkers permanently gain damage whenever they are dealt damage, and Doom Knights(?) gain damage over time once they activated their ability. Or the enemy stepped on a power mushroom.
I agree that dominate not working on some enemies should have an explanation somewhere. From memory, I believe only animated weapons are immune.
Overheal is caused by Orc Battle Priests. Whenever they take damage, they heal all other nearby enemies. This healing ignores max HP, allowing them to overheal.
Cursed ground tiles (forgot the exact name) does the same for undead enemies. This affects basically every enemy in the crypt, except for maggots. Death Priests can cause those to appear, but they are also part of the map.
Orc Ice Shaman, as well as totems from gnoll shamans, give an ice shield effect, which is effectively bonus HP. But this one is very visible.
As for healing abilities, besides the 'overheal' ones already mentioned, there are quite a few of them. All priests and shamans have an ability to heal others. Orc drummers can heal a group. Elites with "regeneration" heal over time. Obsidian Golem heals while it is standing in lava. Musquitos heal when attacking. Leeches heal when moving over blood (I think, not quite sure). And enemies can also stand on a green mushroom, of course.
Totems are summoned by certain enemies. They "see" all nearby enemies (except for the summoner) as they are spawned in, and affect all of them no matter how far away they get. Typically they heal them every turn, but gnoll shamans' totems instead give an ice shield. You should almost always kill the totem first. Totem-summoning enemies can only summon them once. However, when changing floors they regain the ability. This also goes for dominated enemies
In the orc fortress, there are also certain pillars on the map that can cast discord on the player, or heal an enemy. They only use it when you are in a fight with an enemy.
I do not understand what you mean with "Anything I throw at any of the monsters only makes them stronger." Which monsters? Are you sure it is your attacks and not something else? Perhaps one of the effects or abilities mentioned above happened?
Finally, A part of the game is figuring out what things are dangerous, and if it is worth using some consumables to overcome that or if it should be avoided altogether.
Figuring out 'the rules'- that is to say, what to expect from certain situations, is a part of that. I'm not saying I disagree with you, but you arent entirely correct either.