I chose not to and to serve him. Nothing bad really happens and it doesn't interfere with the Orc war event. Though make sure ur sanity level is above 100, otherwise when you do the lewd scene with him, it's game over. Not literally.
