This is a really fun idea and excellent proof of concept. I was skeptical of the idea and within the first 2 puzzles I was instantly sold.
Unfortunately halfway through the second stage I entered a puzzle and the game stopped accepting any inputs from me. This happened right after acquiring the "Golden Key" item, so possibly has something to do with that?