Seems like the lobby is bugged with 5 or more players. To make it work:
* have the leader set the max players before anyone else connects
* No one ready up until all have joined the lobby
* no one leave the lobby, if anyone leaves, recreate the lobby