You need to see the fence only once, to go to the fence with him you need to have 40 likes and talk to him.
