You can get it much later, in the secret quest that starts after defeating the boss at solitude.
I'm just curious, how much later? I did defeat the boss at solitude in two of the games I played, even defeating Tyran after that, and I am still trying to get the key...
