What does the daily report say?
Your mansion is dirty -> clean it
Your mansion is overcrowded -> buy more beds (even better, give the girls a personal room)
Not enough food -> buy more food
Nudity rule -> only Pervert / Sex-crazed girls with high lewdness are fine with it
High lust & no sex since 5+ days -> let them fuck!
Pregnancy -> improve their health, but there is still a chance that their stress builds up
Strict headgirl -> for a few days tell her to be kind with your slaves
Cows -> pay and improve the farm
Magic toxicity -> stop giving the girls potions for a while
Girls fight each others -> improve their relationship meter (the fastest way is if they have sex together, I think)
Some jobs may increase the stress too, but you said you let them rest.
Some items (Sundresses, Miko Outifits) reduce the stress; other items (Pet Suits, Slave Collars, Handcuffs) have the opposite effect.
Stress builds up because of fights too. Check its meter when outside (move the mouse pointer over it).