I actually very much agree with this. It did trip me up, but it actually makes sense. Of course he's not happy you're brushing aside the thing he prides himself on to have an extremely awkward conversation, but if you don't have that conversation there's no way for him to realize how hurt you are that you've drifted so far apart.