The challenge roll system is so strange. I roll above a 1, so I "succeed", but I don't "win". If I don't win, I still fail despite having succeeded.
In order to "win" I add my wanderer's stat value to the actual value of the die plus or minus the die value's modifier. So instead of adding a random value ranging from +1 to +6 to stat value, you're actually rolling a 1d6 with faces that say
[Automatic fail], [+2 and Lose 1 HP], [+1], [+3], [+5], [+7]
???