Is there enough food in your mansion? What job are your slaves assigned to?
(I never remember if, when the mansion is dirty, your slaves are only more stressed or lose healthy points too.)
Anyway, please tell which version you are playing and on what OS. Also, if you could share a save, it would be easier for the devs to find the bug.