I think your sexual experience needs to high, & you need to do the quests the brother bartender gives, so the madame can get her eyes on you first. Not sure of Exact requirements but after that I got sent to the madame to be "examined" by her when asking about a job.
Depending on stats she may reject you for being "too much" or "not enough. Not certain what exactly triggers each reaction. Lewd fame seemingly has nothing to do with it, as i got in with none.