I see. That does sound pretty confusing, yeah. Especially if you try using non-"standard" controllers like the N64 one. I've always been a KB&M enthusiast myself, so I don't have a lot of experience with controllers, and I own only generic x-box-like ones, which realistically cover my needs of occasional gaming on the TV.
Your project sounds really cool, and while it's probably a godsent for people who want to be in full control of controller bindings, unfortunately I think it's is too niche to reach a broader audience. For most people, an xbox controller IS all they need for gaming.
That being said, I think it's still important for projects like yours to exist for those who need it, because more accessibility is always a great thing, and games should be as accessible to as many people as possible. That's why I try to give as many options and key rebind settings as possible. In my current project I have 4 input methods that can be used independently and are completely sufficient to play the game and access all functions - those are Mouse, Keyboard, Controller and Touch.
And having gone to all the trouble to implement all these, I can see why some developers don't want to bother with input remapping and even supporting controllers/touch input - it's a lot of work for something that not a lot of people use (touch gestures in particular are a nightmare). For these games I'm glad tools like JoyToKey, AHK, XBMC and Steam input exist, even if I'm not using them, they're making someone else's gaming experience that much better and that's awesome.