You know, that actually makes sense and for Feliar stats maybe his endurance, charisma and intelligence are around 50? Ok think it this way, during the sex part with Nameless there's a charisma test and you need 25 to passed, you don't resist his powers but, you are able to resist a bit and don't eagerly jump at his log ;)
And since what he did is, from my perspective the magic equivalent of a love potion I think that a high charisma and intelligence will likely help you stay focused if you drink a love potion, you know, thanks to the stats test you can actually measure world.
Let's get real, you need 16 of perception to hunt whit the bow a deer that's lazily distracted, but with 20 you still have trouble fighting a vulture.
During the picnic with Caleb you need 25 of agility to catch him and he tells you that you are as fast as a werewolf (or as a chubby one at least ;) And with 25 of charisma you have 100% of charm without gear boost so I said that 40 or 50 of charisma plus 16 or more of intelligence will be enough for Feliar to be the way hi is.
Or could it be your theory(which I like most) of him being Vicoria's ex and have resistance to the aphrodisiac. But the again if you think about the stats, maybe with 50 of appearance MC will be like the classic anime character that enters a room says "hi" whit an innocent ;) smile and people get nose bleeds or heart attacks. Even more with 75 or 80 he could be worshipped as the God of beauty lol.