He have to like you. When he likes you enough then when leaving the shop he will ask you about meeting him in the bar.
Doing "Lost Friend" quest gives you point. During the sickness event when he want to pay you for medicine refusing gives you points. There is a talk option in the shop. There's a few random situation where you can gain points.
Just remember that using Love Potion lower those points, so it's harder to get the bar meeting scene.