As far as I know, playing the Alpha 0.18.2 that's available, the magic attacks I've tried from staff and wand all require you to use the targeting 'orb' on a body part of the enemy. If you can't see it, the orb is probably inside the character instead of on the 'surface' of it. It can take a bit to get it just right even for myself, but so far that's what I've noticed. (this is the same for normal weapon attacks as well not just 'magic' related ones)