Have you taken him on walks at night? If you offer to take him he'll first joke about the cum-covered wall in the inner room in the barn.
I think you also need to find Rose talking to Caleb through the wall of the barn, but could be wrong about that.