That means you can't hang out with them yet. What do the tasks say in the phone menu? You need to level the gems up in order to hang out with them and unlock sex scenes. The sex button is used once you unlock previous sex scenes.
