Well, the adventures ARE balanced assuming that you'll be mainly rolling your non-d4 traits with only the occasional d4 in weird circumstances. They assume you'll be creating a character reasonably able to take on the challenges of the adventure.
Thus, it's not exactly a problem to come up with traits you don't use often – it won't break the game mechanics – but it will make the game a lot harder.