I loved the idea of your past actions becoming what you have to respond to in the future. My game utilized the concept as well, but not in a rhythm focused way like you.
I think something that could add a lot of depth to the game would be to increase the BPM over time. Then starting off the BPM slow could make the game easier to get into for non-musical people, and then kicking up the BPM can reward people who love the game.