Totally understandable, and its not even just the players that suffer from magic either. 9/10 of the times i hire the fire mage upstairs in the tavern, she will either stunlock me with the enemy with her magic or kill herself with her fireballs trying to take out enemies. it's quite hilarious to watch but i'd like to not hire her a million times because she's that unreliable lol
i think the main problem is magic seems to be designed only for offense, and you (for some reason beyond me) can't take acolyte traits or divine items if you choose magic. it would be so fun to setup fights with a fireball, and then heal your companions with divine items or mend soul ability, and gives you a bit more options in regards to magic. but i imagine this design choice was based off the lore around the saint, and the followers of the saint.
at the very least I'd like to see some defensive magic options that aren't just abilities that block enemies paths, that'd definitely make it more viable at lower stat levels! (like Cocona's necromancer shield she uses for example)