As of now, the only thing that matters is how far your Gatherer's Lodges and Barracks are from activities like Imp Camps, Abandoned Villages, and Divine Offerings, since your gatherers and soldiers have to walk back and forth. Lumbermill distance does not matter since Wood and Gold are gathered on the day tick.