Sorry for that. I did some adjustments on version 0.24.1 for these problems.
If your charisma just decreases, it's probably due to bad results of the locker room H-minigame minigame. But it's adjusted now to be more balanced.
If your charisma gets negative, I left a fix calculation so it turn positive.
Thank you for the report!