The importer is picky on making sure that colors match across the board. If you are off or missing colors in the sprites.png that don't exist in the colors.png, they are ignored. I will look into a way to help debug import via the log so you can have more visibility into what is happening when everything is parsed
This threw me off for a couple hours as well. It would help to have a blurb about this in the document section related to loading sprites. Maybe just something that mentions the issue and points the reader back to the color map section for further information.