Really tough! I think if the controls were mixed up but didn't change all the time it would be a little less of a memorization game, unless that's what you're going for! Cool concept either way.
If you have time, please check out the game I worked on for the Jam called "Foobar". Thanks!