This is a fun concept! One way that I feel can fix some of the things mentioned below would be to remove the limited dice roll and just count the number of times rolled overall as a 'score' to show at the end of the game. Overall a really neat game!
Fun fact: I also used the same dice animation asset in my game haha