To explain the design side of it, hitting the Green zone gives a bonus equal to 3 seconds, Yellow gives 1 second, and Red removes 1 second. Of course, if you are not hitting Perfect consistently, I understand why it can feel like “what is the point?” My intention was simply to reward players who perform better.
The original idea behind the system was to make scoring feel a bit like rhythm games, where hitting Bad might cost score but not necessarily break Combo, for example. During development, I decided to put the Combo idea on hold until I can find a better way to implement it later.
From my own testing, it is possible to clear the minigames without buying upgrades if you play carefully enough, so I felt the balance was acceptable. My thought was that if the game feels too difficult, players could try buying upgrades first. Once all upgrades are unlocked, the game honestly becomes much easier.
That said, I do agree that having Bad (Red zone) and running out of time give the same penalty should not happen. I will definitely fix that.