Some equipment cause the sanity to drop,like the witch's amulet and mysterious signet each one drop your sanity by 20 point.
Oh, I didn't notice that. I had both equipped. Thanks