If you implement a food rotting system maybe think about a compost barrel that way there is a need for compost put in pots for planting. It can be a required item to build said plots. I love the plots but it would explain where the dirt comes from in game.
Either that or add compost to dirt and yeild twice as much food.