That's just the way it works. If a spirit lingers in your body for a long time, it can't be removed automatically, even if there's a button. It's best to let it linger for a little while.
