It's not a bug, you have to find new ways to improve your relationship with him. Different actions have different caps for how high they can raise the affection meter.
