I've had the same problem. Only if harvesting herbs then toxic berries was the priority. But at the same time they should pick up an axe lying on the ground and start chopping trees for food and wood, and harvesting more filling food first then herbs then toxic berries.