Figured it out. After visiting the tavern so many times, you have to go to sleep and end the day. The challenge is completed successfully when you wake up the next morning. I also had the same question ahaha.
That seems to be the case for all challenges. They complete after ending the day if you meet the conditions.