Robin will leave you if you do any slaver stuff with him in your party, since he was a slave. You must missed the notification that says he leaved you for being a slaver.
