Most choices gave affection points, even ones that didn’t seem like a big deal. Maybe you missed a lot early on? The scene I’m assuming you’re talking about is meant to be hard to get because of the context surrounding it, but it’s only one H-scene and doesn’t affect the story, so I wouldn’t sweat it too much.