This happened to me twice as well. The second time I saw that it happened when I reassigned an "idle" villager to destroy a wall. Since the game often reads hunters as idle, it assigned a villager with food in their hands to destroy the wall. The villager was headed to the town center with the food- instead of depositing the food, the villager destroyed the town center. It's game breaking, as doggg91 says.