Dont get negative happiness as 0 to -9 happiness have a -50% pop growth debuff, and anything below stops population growth entirely.
Also you might want to settle near food resources like banana and sheep, river can buff nearby farms to have +1 food bonuses after researching civil service.