Yeah that can happen... You can reset with backspace.
I'm not sure why it can happen because the collider go all around the buildings. Might be that it is still interpolating movement, I'll have a look into it and see if changing that fixes it.
