You don't need high stats but maybe you have to progress more in the main quest
Nah, story progress has little to do with it, cuz I managed to make it work before even entering the desert, and certainly well before doing the first dungeon.
