You have to catch him watching you sleeping in the temple twice, then confront him about it, then call him to sleep with you once, also after he asks if he should expect an invitation say no because then whenever you go to sleep he comes to your room at the temple(put it here because this part is confusing at least to me), the second time he comes to stay the night you'll get the option to have sex with him after that you get the love point