Have you ever used Love potion on him? It makes him mad at you and left your house. But if you haven't, there would be an option to talk to him while you stay in your house.
Yeah i think i gave him the love potion
If he left you, there no way to bring him back. I suggest you to go back to previous save, in case you want him.
