As a drummer myself, I love this concept! I was great fun to get into. It reminds me of a minigame from Rhythm Heaven in an obscure way, where the music builds up the better you do. I love it!
However, I, like most other commenters, found it difficult to follow the rhythm to click to. I was lost until I was following the yes/ no in the corner. I think using a visual is more than fine and could certainly be made into a main part of the game - a pulse at the bottom, animate the drummer, or have the drum flash a different colour, for example. I assume the idea was to click in time with snare drum, but the window of time where you click felt a little out of time. I would also be nice to have a consistent art style.
However, this is definitely the game I've had the most fun with so far. Good job team!