I was really confused by this too. After finding this comment & trying Violet -> Remove repeatedly, I finally noticed that the Armor bar in the top left takes damage, and it's when it drops to 0 that Violet is actually removed. I hadn't noticed that before. I had sometimes been able to get Violet removed via Nel's events or other methods, but nothing easily repeatable. And maybe I'm way over-leveled (so Violet has more health being used as armor), but in my case it took a *lot* more than 3-4 attempts.