After they produce their own food I can ignore them for chunks at a time. Once you research metal you have to wait quite a while. They'll upgrade every single house they have before moving to any metal based tech. It's not like the world will change much without you. The worst is when the ones with axes die out, but if they have farms and seaweed some will always survive.