Absolutely agree, they're a bit social when they are waiting for crops to grow. Having multiple workers has a lot more advantages once there are more farm plots available as it allows them to spread out better.
EDIT: I just pushed out an update that overhauls how your workers behave. This should ensure they do not waste your money and all handle their own plots now even on small-scale farms.