Awesome idea, a little on the vain of Baba is you.
I think the levels felt randomly generated which kind of defeats the purpose of a puzzle game. I was able to get to a score of 20 by just pressing one key one time, since the rules allowed me.
I would love to see this built unto a more fleshed out game!