I experienced the same thing...
Other than that, You can clip through the floors at the tower.
and when you do that it bugs him out and he cant attack you
