You're probably not the only person who thought that, but I believe the implication is that Yuu only being attracted to men is why all his past relationships with women have failed, and that he only got into those relationships because they asked him out and he really struggles with saying no.
