The hearts are affection points, so you must choose a character and focus on him, ignoring the other.