i have done several playthroughs of the game and have not come across a way to remove it.
i believe its been put in to make players think about their choices, as later on having the higher charisma makes it alot easier.
but i will post on here again if i find any info on it.