A full spells build without Incantation Headband?? Can't say I've ever thought to try that. Hell, I use Incantation Headband even on some primary physical builds because it allows me to sometimes passively rebuy utility spells without wasting actions on breaking purple gems. I definitely always use it on primary spell builds. I consider it easily the best item in the game. (Granted, I only have the free version, not the Patreon version.)
I mean, obviously your damage with this build is mainly physical. But with every single skill being a magic skill, I think the extra purple gems are definitely worth the +3 MAG being minimally effective.