I think it has potential. You could make some interesting puzzles with that mechanic. Try fleshing out the prototype a little and get more feedback.
