Love the concept! I think it just needs some added elements to mix it up and prevent tedium. A win condition, maybe throw in "fake roll requests" that you need to look for and are not supposed to roll for, maybe make the modifiers get larger over time and add a way to unlock +3/+5/+10 modifiers to make it easier to keep up with them. Just some ideas to further build on this concept. Otherwise, great job!