You should start by visiting the blacksmith, then confront Duncan about your father.
After this, you have to spy on him, and keep on spying.
That is usually the part most people miss.
After that, just ignore the barbarian, read the diary,rescue Duncan alone and confront him.
That should do it.