Young padawan, If you want a normal gameplay experience, you may simply ignore the fetish exp side. If you want a harder game, than read well.
The Fetish Exp is a reward for losing in the right way, and can either develop naturally by these lovely girls doing something harmless such as grinding against you, whispering sweet nothings, or spitting some harmless fluids upon your body. Then there is satisfying the girls, doing the devils tango, feeding her your male milk, or being pampered by some soft love pillows. The more you engage with these fetishes, the more your willpower to oppose these fine women will diminish and instead try to satisfy will grow. 
While these numbers are nothing to scoff at, there are thresholds when your mind begins to adapt. Some times that mental correction will make it to where once you were fearful of monsters, to an instinct that will make you think of her as wife material. Libido is your sex drive, making it to where what once was a rough touch from these fine lady's into one that sooths your soul, letting her know you are ready to serve. While you are right that with a high Libido, it will allow one to ascend to being able to send your flames of passion as a form of offence, it also makes said feelings more pronounced.
(Silly bit aside, Fetish Exp will make it to were a woman touches you, your lust bar will go up. The worse the trait/the more fetish Exp you have, the more lust damage you will take, and potentially force a stun that skips your action. Libido is kind of a multiplier of said lust damage, think a lust attack that deals 20 damage, and having a libido of 5 as a times .5 multi, thus the end lust damage taken is 10. While having a high (16 for me) makes it to where your pyi attacks do more damage( 20x1.6=32), you also take the same amount of lust damage. Hope this helps.)