A nice core of a game! it'd be fun to see how more decision making could be incorporated. One way might be if there were several dice with different numbers on them you could choose.
Funnily enough this reminds me a little bit of a pretty unique level in Wario Land 4: https://strategywiki.org/wiki/Wario_Land_4/The_Big_Board