Okay it was probably my own fault, I had been assigning them to gathering but maybe they only gather one resource and then go back to idle.
