I think this is a really neat mechanic, but I did struggle to keep track of what the player could interact with, without more key visuals. There is a pattern I know called a Bayer pattern that might work really well here to colour in the background with.