Just figured out a fix, its a result of something going wrong during the connection process, where after you fail to join the first time, the server still thinks you're in it, so restarting the server and rejoining worked to fix the problem.
