All non-red buttons are connected to the grey crystal and make it appear on a separate trigger. Which means that all non-red buttons need to be turned back to "off" to get the grey crystal to disappear.
To make it a bit easier for the players, since they're locked into this area if they can't figure it out, I changed the blue, yellow, and green buttons to stay down while they're active now, and come back up when you deactivate them. This should make it easier to deactivate the grey crystal, since you can see which buttons are active.