I think you can go back and re-enter the manor in this specific case, have you tried that yet?
You can indeed!
