I think it was the tall blue windmill, with two or three squares of room above it on the grid. If the worker tried to go over the windmill they would disappear behind it and never come out, maybe they were still collecting bread and I just didn't realize but it confused me all the same when three of my workers just vanished lol. (I was using the windows version of the game if that's relevant)
Again though, everything else was great and I think an expanded version of this could be really fun :)