You have to observe the workers and move about in a way which doesn't allow cross their path. You can however get as close as you want to them from behind. Just plan out he route and stop trying to speedrun the game
