Have you checked in front of the castle if there is a guard with an !-icon above his head?
There is a guard infront of the castle but without the !Icon.
Did you use the fast travel system at least once after unlocking all requirements for the next event step of Grey?
