You have to do some misc. tasks before being able to go to the elevator from time to time. Depending when the moments to go down occur they can be as simple as doing what the MC said they wanted to do, finish up doing something for another character, looking your bedroom for an interact-able you might have missed involving either of these things (like the walkie talkies in one instance for me because I had started seiko and forgot), or talk to someone and finish up their small interaction moment. I only had one or two sequence breaking moments with Aysha that prevented me from going down once until I got another character who ALSO needed me to go down to remedy the interaction :)
TL;DR the "Not Yet" I believe is meant to be a 'you still have some tasks in motion' I believe, but I think a more explicit 'you need to do something involving [X]' would make it easier for some folks.