There's no way to fight the wizard EVER if you choose to give him the crystal even once - dying and respawning after doing so leaves you unable to shoot fireballs.
This is even more of a problem than it seems considering that the 'give crystal' option is the top one, so it's what you'll hit if you mash spacebar to get through the dialogue.