It literally takes the hex value for the chameleon color, subtracts the background color, and if the difference is less than 20% you're hidden enough.
Unfortunately this has weird results sometimes, and even when you clearly see the chameleon he is considered hidden, or when he looks pretty close, he's way off.

If I expand on the game I will definitely rework the color comparison and reduce the number of colors. :) Another thought I had would be to hand pick the colors, so I would use "moss green" to color any moss in the game, for example.