uh wait you can get him at diferent times? is him joining when he finds your camp the later or the earlier?
he finds your camp after you interact with him enough, so if you just ignore him you can get him a lil later, as I understand it
