This would be a great game if there was an explanation of the game mechanics. Just got killed by a monster in one hit. His max damage of any of his abilities or weapons was 28. I had 35 hit points. Turns out in that in this game 28>35. In another game, I found out the hard way that some monsters are immune to dominate, but there is no information about that in the monster's info and the explanation of dominate I saw did not mention that some monsters are immune.   And what is "overheal?" none of the monsters seem to have a heal ability. There is no totem. Anything I throw at any of the monsters only makes them stronger. Again no explanation. You can't play a game with any skill if the rules are are a mystery.

I agree that there are several mechanics that are poorly explained. However, some things are probably due to something you missed. Things are easily missed, of course. A log would be pretty cool.

Was there only a single enemy? Did you have the "unstable", "grappled", "netted"(?) or "discord" status effect? Unstable is caused by terrain, grappled and discord are abilities that some enemies have. Also, when an enemy that knocks you back into a wall, you get a "crunch". I believe each of these multiply damage by 1.5.
Another posibility is that the enemy gained damage somehow. They can get the berserk status, and some enemies have other ways of increasing their damage: Orc Berserkers permanently gain damage whenever they are dealt damage, and Doom Knights(?) gain damage over time once they activated their ability. Or the enemy stepped on a power mushroom.

I agree that dominate not working on some enemies should have an explanation somewhere. From memory, I believe only animated weapons are immune.

Overheal is caused by Orc Battle Priests. Whenever they take damage, they heal all other nearby enemies. This healing ignores max HP, allowing them to overheal.
Cursed ground tiles (forgot the exact name) does the same for undead enemies. This affects basically every enemy in the crypt, except for maggots. Death Priests can cause those to appear, but they are also part of the map.

Orc Ice Shaman, as well as totems from gnoll shamans, give an ice shield effect, which is effectively bonus HP. But this one is very visible.

As for healing abilities, besides the 'overheal' ones already mentioned, there are quite a few of them. All priests and shamans have an ability to heal others. Orc drummers can heal a group. Elites with "regeneration" heal over time. Obsidian Golem heals while it is standing in lava. Musquitos heal when attacking. Leeches heal when moving over blood (I think, not quite sure). And enemies can also stand on a green mushroom, of course.

Totems are summoned by certain enemies. They "see" all nearby enemies (except for the summoner) as they are spawned in, and affect all of them no matter how far away they get. Typically they heal them every turn, but gnoll shamans' totems instead give an ice shield. You should almost always kill the totem first. Totem-summoning enemies can only summon them once. However, when changing floors they regain the ability. This also goes for dominated enemies

In the orc fortress, there are also certain pillars on the map that can cast discord on the player, or heal an enemy. They only use it when you are in a fight with an enemy. 

I do not understand what you mean with "Anything I throw at any of the monsters only makes them stronger." Which monsters? Are you sure it is your attacks and not something else? Perhaps one of the effects or abilities mentioned above happened?

Finally, A part of the game is figuring out what things are dangerous, and if it is worth using some consumables to overcome that or if it should be avoided altogether.
Figuring out 'the rules'- that is to say, what to expect from certain situations, is a part of that. I'm not saying I disagree with you, but you arent entirely correct either.