Getting the option for "Yes - but not right away" means affection is not high enough. The usual culprit here tends to be the birthday choice because it changes depending on current affection level and if affection is not high enough by this point you will lose out on the chance for another affection boost.
The easiest to get the right end is to simply replay with the affectionate trait. Alternatively you can replay with the guide and ensure you have 9 affection points by the end of the game.