I do have the same problem, it is hard to remember what color belongs to which button. Also the esc key to close the controls is the same as closing the fullscreen. ;)
Maybe fitting music to the beat would have also helped a little.
But still I think that its a fun game. Good job! :)