I figured it out, just take wo you want to leave and take them somewhere that's not your village, like the dungeon or spider cave after it's been cleared. Nothing gets affected and you have to feed less people. When you want them back just travel over there again, theylll be healthy