I think you should be a bottom. His visits happened with me, but after I told him to leave my hut in Bareshade when he snuck in without invitation, my likes with him decreased to 1 (it wasn't high to begin, lower than 10, but it was enough to get s*x scenes with him). Telling him to leave was necessary to create the route so you can get his first and only heart, which I did get. However, it also stopped his night visits.
I am going to try and raise the Bandits Moral to see if I can make him return, because mine is in the negatives lol