I just gave it mana blade or a different abillity or do you mean the occasional swap back?
The swap back, though I'd imagine you planned for that in your build so it would be less annoying
I gave one skimander a manablade so it never casts and the other got a healing tome or something to replace their swap ability, with swap back I mean when the enemy does the cast thing and I lose dps as the speed bonus was position sensitive