Interesting idea, and fun to work with the amount of dice you receive. Love the visuals, these dice are my friends now.
To avoid having outcomes where you realize that you can't solve the puzzle with the dice you have, having a way for any dice amount to solve the puzzle would keep the player from needing to reroll.