I'm copying my response from earlier:
The hearts on a girl's overview screen do not indicate the precise number of affection points you have with her. Instead, a scale is used; e.g. having 2 hearts on Dalia's screen means that you have 4+ points with her, 3 hearts means you have 7+ points, and so on. The scales are different for each girl and they seem to be somewhat arbitrary.