As long as you do not take him while you do slaver stuff, he will never leave. No crpg companion 'oh I know you did bad even though I couldn't have found out' bullshit.
