So, what happens if you try to carry more?
You either don't have enough space in your pockets or you can't handle that much crystal energy near your body. I didn't really write a lore reason for why MAGI can only carry a certain number of crystals, it's just an abstraction for game balance that you can create as much or as little narrative justification for as you like. It's a resource you must spend and recover, but the way you access that resource in the fiction is intended to be loose and varied so that you can pull one out of your pocket, break a nearby sign, etc. With the Salvage action in the play guide allowing you to roll dice to replenish your supply.
I realize it's a bit abstract, but you should treat it like any other LUMEN-based energy resource with a fictional representation as an object you can hold a set number of.