Congratulation!
The special mode is only a mode that lets you play music freely. It's just a little bonus, not really worth putting hours of struggling to unlock it. (although it's ironically easier to get on the hard mode, but this mode is, well, true to its name)
The green level is indeed the hardest one. It has (almost) true random patterns, which can be treacherous. Lots of back-beats, and "3 fourth" gaps.
Edit: Wait, I've been tricked by my own colorblindness! (… or rather my flawed long-term memory). Green isn't actually the last level, it's red/pink! This one is supposed to be the hardest! So forget what I've said. Maybe the difficulty comes from the instruments used on this part, as you've said.
Thank you for sticking to the game despite its difficulty!