Hey, that's a pretty cool idea. I'm actually changing up that menu quite a bit in the next update though, to make it more clear. I got rid of the affection points completely and have scenes unlocked based on what previous scenes and conversations you've had with a character. And instead of showing a number of affection points on that screen, it'll show the completion percentage. That should help quite a bit with who you have completed content for.
Thanks for the kind words!