I thought the backlash was because of the dream/memory segments that depicted other men?
Although this still doesn't count as NTR since the girls aren't being shared or stolen from the MC. Since the scenes depicted were part of the past.
The only NTR was the netori. MC taking Mary(?) From her ex husband.