Here a lot depends on the stage of the game. At one stage you can remove it in 3-4 attempts (you need to have enough health), at other stages this can be done with the help of other characters. For example, you can go visit and fight with a pet lover, or lend the body to a ghost in very reliable (no) armor (and then return it with the help of Nel).
I was really confused by this too. After finding this comment & trying Violet -> Remove repeatedly, I finally noticed that the Armor bar in the top left takes damage, and it's when it drops to 0 that Violet is actually removed. I hadn't noticed that before. I had sometimes been able to get Violet removed via Nel's events or other methods, but nothing easily repeatable. And maybe I'm way over-leveled (so Violet has more health being used as armor), but in my case it took a *lot* more than 3-4 attempts.