Yeah, I actually thought of something like that as well !
Like with the rogue that jumps higher than others, or the warrior that could break boulders ... but as you say the randomness would make it a bit frustrating maybe.