The base idea definitely did end up working better than I thought. Especially with limiting the dice to 1, 2 and 3, which helped control the randomness.
I do believe undoing dice would 100% work, but would need to make sure you can't undo dice from previous turns.