A rhythm game centered entirely on holding keys is, I think, new to me and definitely fits the theme very well! The fundamental ding-ding-ding-PRESS pattern is also quite fun!
I definitely think that it feels a bit idle during the times when you hold the key for a long period of time. Usually in a rhythm game, you aren't necessarily doing something all the time, but you are usually doing stuff at least at a pretty decent pace.
It's actually pretty hard (at least for me--and this could be skill issue) to feel the rhythm in this game, and I think that's partially due to these long sections, but also partially due to one other factor: the ding-ding-ding before each action we do is very prominent, and it's somewhat difficult to feel the rhythm of the rest of the track with respect to the actions we're taking due to how prominent they are.
So one thing you could try is to turn off the ding sounds, and see how it feels to try to feel the rhythm based on the musical elements that are in the track. I guess you still probably want some sort of audio cue, but I do think that would help make the times in the track where there are no dings for seconds on end feel more integrated with the parts where we do the taps.
The other thing to try would be to make the game more active. If we had to do things more often, I think the game would be a lot more engaging and, at least for me, more fun. I think that there really is something cool about having to time when you release a key, as it's not something I think about a lot when playing games but it is something I think about all the time when designing them. Focusing in on that aspect of the keypress is interesting to me! And I think having it happen more often would make it more fun.
And to be sure I am not a speed maximalist when it comes to rhythm games. I am definitely more interested in unique rhythms (e.g. triplets / other polyrhythms, weird time signatures, etc) than pure speed, and I think those kinds of things would work here too! If I'm not wrong, there was already at least some variety with where the beats landed that we had to do, and I think those mixups were fun! And I think even more like that would be great too.
As one final note: very good job making a complete, polished product. Despite my gripes with the gameplay, everything works, all the art is there and looks awesome, and the music sounds great.