If you have not played "Osu! Tatakae! Ouendan" / "Elite Beat Agents", I highly recommend that you check it out and see if it gives you some inspiration. Specifically, look at how they use contracting circles to show precisely when you're supposed to press, and look at how that timing matches the music.
The problem I have with your version is that I'm not feeling the music; I'm just waiting for a button to appear on screen and then pressing it as fast as I can. As a player I need some way of understanding exactly when the button press is supposed to be, and being scored based on how precisely I've hit that mark forces me to engage with the music.