The elf mage would have told you to find someone with a sword. It's not required to find the elf mage to progress.
