Was designing rotations fun or frustrating?
It definitely wasn't frustrating. It was more interesting. I liked figuring out the rules. It's unclear from the prototype whether it would stay interesting after extended gameplay, but the prototype was good.
Did failures feel fair and understandable?
I actually felt a little cheated, since the game withheld essential abilities that were required to defeat the first enemy. Understanding how the game worked was not enough: you also had to wait for the game to (arbitrarily, it felt) decide that you were ready to receive an essential ability.
I would have preferred receiving an ability as a reward for defeating a lesser enemy, or after having earned enough experience points to purchase the new ability. That way, I could strategize about what to purchase, and look forward to / enjoy the purchase. The new ability could make the next enemy beatable, or it could accelerate the rate of experience gain, making further purchases possible.
Did you feel encouraged to tweak and retry?
Yes, I did. It's essential to the game.
I only needed the first hint: once I made the first change to my ability sequence, I no longer needed / wanted the tutorial.
However, an explanation of the abilities (both player and enemy) would have been helpful (e.g. the effect and the cooldown period), as well as an explanation of the combat algorithm (e.g. when you have multiple abilities that could be used, which one will the combat algorithm select?).
Would you play a full game built around this system?
Yes. It's unclear from the prototype whether it would eventually get boring, but it's off to a good start! With some additional mechanics, like a skill tree, and monsters with various abilities, it could be quite fun.