When additional players join, the host tells them to connect to each other. Sometimes (usually due to complicated NAT setup) this doesn’t work out.
On Steam, having everyone rejoin the lobby (or, worst case, restart Steam) usually fixes this.
Off-Steam, you can try having people join in a different order, but might have to use one or other VPN software to bypass this problem.