You have to visit the spirit shrine when your character is in control. So, struggle until you get control tokens, then, before they wear off, go to the spirit shrine and get rid of that pesky ghost.
