Nice concept, but needs balancing.
Blacksmiths and temples are way too powerful, especially when spammed everywhere. These scale linearly and then multiplies with workers, which means total production is square of workers. This is way too powerful. Maybe they could be changed to research building, and you buy researches to power up workers.
Also, due to the central limit theorem, the factor of dice soon becomes irrelevant, when you have many more workers.