Okay it was probably my own fault, I had been assigning them to gathering but maybe they only gather one resource and then go back to idle.
Yes, if you manually assign them to gather one resource, they will gather it and go back to idle. But if you assign them to the gathering hut they will automatically go and collect every resource on the map.
