I know, but the point is quality rather than quantity. Your packpack space is limited and the buff that gems provide has a upper limit, which will quickly become ignorable when you're in endless run. Other builds (black manastone, assassin dagger) have no such upper limits and therefore possible for endless. I'm suggesting something similar to remove gem's limit.