We dont know yet what will happen at the end of the game... maybe you end up with him if you service him? Or he maybe gets a change of heart......?
Just choose what you want, I don't really think Hyao would punish the people (that much) at the end..
