Ah, if you declined, you've missed your chance. You'll have to go back to a previous save and ask him again.
