Numbers are not words, they are symbols. Not having numbers constraints the possible games that could be made too much imo. Most games in the reference section of the briefing post have some sort of numeric representation, and I believe there is a reason for that.
I do understand that constraints are a way to canalize creativity in a specific direction, but if the constraints are too rigid, or too many, you end up with only a few possible games that could be made.
Examples of games that can't be made without using numbers are: (most) Incremental games, or games focused on Resource Management. Because you can represent 5 pieces of wood using icons, but you can't represent 100 pieces of wood using only icons. I mean... you can, but them the player experience would become worse, not better, and this kinda goes against the whole idea of this game jam.

