If you are staying with the current model, I have a few suggestions I think could help with clarity.
-Distinct border colors based on the relevant character.
-(if doing the previous suggestion): a key/legend on the side. Picture-Name-Color. So you can see at a glance rather than needing to search around
-replace the red/green with a check mark in the center. Perhaps a lock icon for the currently black swuares
