After far too much procrastinating I've finally got the dithering working in-engine!
I was confused for the longest time because most of the writing on the subject that I could find was as clear as mud. Lots of formulae thrown around without any explanation and ominous warnings about how 'you can't do this'. So, instead of trying to copy code I didn't understand, I broke the process down into smaller steps and added them to the screen shader one at a time.
All of these steps use a 16x1 pixel texture as the palette:
- Create a luminance-based quantizer using the first part of this shader. This just arranges all the colours from darkest to brightest and bands them together based on their relative brightness. This gives us bad colour reproduction and bad value error.
- Now have each pixel choose it's nearest colour in the palette instead grouping them by luminance. In my case this means for-looping over the 16 colours. This turns the 1D colour space of the first step into a 3D colour space. This gives us better colour reproduction but still with bad value error.
- Add tromero's dithering matrix as a noise texture over the whole image before quantising. The noise randomly pushes some amount of pixels out their bin and into a nearby one. The shader is now a slow and bloated ordered ditherer. The colour reproduction is good and the value error is low.
At this point I was sure I'd have to come up with a step 4, since applying 1D noise on to a 3D colour space was bound to harm the colour reproduction, but it actually looks good! There is some weird fringing going on around the edges and the image is very noisy but I quite like how that looks for a horror game.
So that's it, a big long technical post. Hopefully next time I can talk about actually putting some game into my game.