I'm thinking on giving myself a bit of a challenge, should I?

if you already played through all or most of it in Wimborn then sure. 

you start out in Frostford where you cant buy armor, the enemies will eat you if you go outside, and there is no Mage Guild like there is in Wimborn.

It's possible to do it but the shops in frostford give very little help, guild contracts can be hard to finish, and you have to build up your mansion and economy for quite a while before launching out.