Nice concept, but I had a really hard time finding most of the answers. At first, I thought the compass would change direction (like north on the right, south up, etc.).But even when I realized the hint was the colors, I was still wrong about half the time because north and south looked almost the same shade of green. But the idea was great, and it could be a really nice puzzle game with a bit more polish.
I completed it in 267 seconds.
Edit: After posting this comment, I replayed it and noticed that all the colors had change. I managed to finish it easily with a time of 103 seconds. So I guess I was just unlucky the first time with the colors. Be careful with RNG like that. :-)