Thanks man. It hasn't solved the problem, but has helped to narrow down the cause. Now we know it is not caused by connection, but rather by the game freezing. It just stops and it doesn't let you Alt+F4 it, so you have to finish the task to close it. After 30 seconds - 1 minute (maybe changing the timeout value makes it longer, so maybe it freezes until we timeout. Not sure tho) the game comes back to life, but we are disconnected from our game and have to continue alone.
It's super annoying, as it happens very often and ruins most of the runs. Sometimes it even happens when we pause the game, so we don't think it is related to the amount of stuff that is happening on screen.