You mean the farm itself, not building next to it? I have tried but maybe misclicked or something, thx I'll try again.
Or do I need to talk with poor farmer again and only then try to upgrade it?
Oh I have found the way how to get Ryan idea for upgrading farm, thx.
Just so other people would know:
)You need to talk to commander Hankin (dude with green hair and delicious cake in his house) his house is house at the right side in Calbridge Stronghold
(Or atleast I think so, do not think that it is the right answer until I remove this note)