When you are on the polyamorous route, day three, after having given them the blueberry juice, you wake up confused and vanilla is not there, just like shadow. There it gives you the option of where to go. If you choose to go to the main room you meet ginger, he is looking for vanilla and had come from a mission. He tells you not to say anything to vanilla and to let him know if you find him. Then if you go to the library it's vanilla and gives you the option to tell him or not.
The most important thing is take off the brooch when you are in the closet, in the start. I dunno if the person who you talk first chance something but I don’t think so