How do you get out of the button dungeon with the new girl?

The grey crystal appears when you activate it with one of the non-red colored switches. You need to turn all non-red buttons you pressed back to "off" to make the crystal disappear.