That's very strange! Does this happen at any point in particular, like saving and loading the game, or placing a building? Will try to see if I can fix it!
This happened to me twice as well. The second time I saw that it happened when I reassigned an "idle" villager to destroy a wall. Since the game often reads hunters as idle, it assigned a villager with food in their hands to destroy the wall. The villager was headed to the town center with the food- instead of depositing the food, the villager destroyed the town center. It's game breaking, as doggg91 says.