I'll be expanding the romance route, so at some point you will be able to have fun with him without having to use the love potion. For now you can talk with him and meet him in the bar for some additional scene, but nothing sexual yet.
I guess I made it a little bit hard to get. Doing a "Lost Friend" quest gives the most points. You can get some when you don't take payment after the incident with sickness. Except of that you just need to talk with him, there's like 6 different topics and some of them can give you points. If you have enough then he will stop you when leaving the shop.