If you are from the house that the girl dislikes, you cannot progress her storyline. Thus, she will never be visiting you in the first place