You probably need more likes. Help him out around the farm, help him with the calves, make the right choices, get enough likes, and eventually he will want to talk about visiting Bareshade. Then you can go. And make sure to talk in the morning when there is time for you all to travel there.