You need to have had seggs with him when you first fought him in the bandit camp.
I'm genuinely surprised I didn't do that in my current save, thank you
