You should be able to remove them from the palettes json in the assets folder. It should be called user_palettes.json, and should have all the ones you created stored there.
I will add a deletion action to the palette window though, to permit in-app palette removal (screenshot attached, not uploaded as of this comment, I will iterate and test it and upload an update asap!)