There is little challenge in walking in circles. It was more a puzzle (remember about the medallion, understand what is needed to defeat ghosts) than a challenge, as ghosts are easy to dodge as they tend to walk into each other and form a small blob of death.
And as it still has a puzzle element, I would not add a skip button. But yes, something must be done with this part of the game as I do see how it can disengage people. Maybe you can make ghosts more interesting instead.