You need to give the fisherman something that will be as sturdy as a stone slab. For example, the culinary masterpieces of certain dwarves. The quest chain begins with Fargrim, and you can get hints within the game itself through quests related to Violet and the overall storyline (the fortune teller, the bartender, icons on the world map).