Have you chosen the option after arriving at your room to decide to explore the place instead of resting? Because if you did everything right, when at the closet with him, he should bring you to his domain.
