I did it randonly on my first gameplay, didn't even knew it was an issue.
I think it happened before I spent time with him at the weekly festival in Bareshade (I remember something about the MC mentioning the new house in town and asking him to come visit anytime, but it can be just my imagination)
From what I saw, you need to already have seen that scene with him and Barrel at the tavern and maybe try giving him gifts. Rose sells an item called top something you can give it to him.