I haven't seen anyone else posting about this big bug yet.
I think this started when I assigned a slave to produce pheromones, then assigned them to regular work after they made 5. From then on I had an ongoing task that wasn't being completed in the mansion. And after about 5 weeks game time I have two ghost workers upgrading the mansion and hunting meat - they take up a work slot and do not gather resources, and they have a blank icon on the work screen. Also one slave that can only rest after I have them a day off of training.
It really got bad after I enslaved a a dozen slaves from a bandit den, then handed a couple for quests, used some to increase abilities, and sold the rest.
This has made my save unplayable.