I managed to stabilize more or less forever by getting my population down to 1 and then giving the crops time to multiply and cover the island. It was a rather cruel narrative until at some point my villager must have found a way to clone themselves up to a stable population of 6-7. Sure it could be that some more seafarers landed ashore, but that's my head cannon and I'm sticking to it.