It's deliberate that you don't deal damage to the ghosts, the intent is to run away from them. You should be able to escape though, I'm sorry it's been giving you trouble. :(
