Unsure if this is the issue you're seeing, but from what I remember when I was doing it I had to do them in order. (e.g. at one point both the, I wanna say 5th and 3rd were still "to do", with the 4th as ????, but I couldn't actually do the 5th until I did the 3rd and 4th. Can't remember if 5th was dinner with Lavish though. I *think* it was, but not positive.) If all the ones prior to the dinner are complete though, I guess I'd just work on progressing all the other Office Harem girls individual stories. (IIRC, another girl pops up during the dinner, so you may have to have progressed far enough in her story)