This is a great idea! Good thing there are reflections of the faces on the board, it helped me a lot.
However, I firmly believe that controls should never be the "difficulty" of a video game. I think that's also the point of a gamejam: to try out original concept.
Overall, good job on your submission!