I had also found myself just watching the game play itself and ignoring the controls. In fact, that's how I tested the game for most of the development (the controls were the last thing I implemented other than the main/help menus).
A lot of the feedback I've received mentioned adding more features and/or controls. I had considered adding more features (something like 6 instead of 3), but I had only really came up with a good 3 while planning, and I was already running out of time with just the three. I still think that adding more would start to clutter the design a bit too much, but I did see during development that there was an issue with giving the player enough to stay engaged.