You could make another character for it and keep your current one
If your respect is lower than -10 (If I recall correctly) he refuses to be your slave.
