A rhythm game masquerading as a boxing game could definitely work, and I like the structure you've put in place here. You built a tutorial, which people often (wrongly) skip in jams, and the timing always felt like it responded correctly to my clicks.
Some suggestions... The music doesn't make sense for the gameplay or theme, however, and the clicks don't seem tied to the music. I'd tackle that first. Second, the location of the rhythm interface makes no sense — you're using the bulk of your screen real estate for flavor rather than gameplay, and it's a waste. A central layout more like Guitar Hero would both work better and let you do more to vary up gameplay (punch left, punch right).
Overall had fun, nice work.