I had to think about your complaint regarding modifiers a bit more - I think I've now got a good idea on how to handle this better:
I'm currently considering adding a "remove modifier" function. This would allow players to sacrifice two cards and in return to remove the modifier. This gives real weight/cost to removing a modifier (player pays 2/3 of a card level up), so players cannot just remove any 'annoying' modifier they come across - but in most cases it prevents players from getting stuck (or needing to grind the same bout many times) if a modifier is too punishing.