I had a really hard time remembering which color beats which while playing.
I think maybe some visual feedback on whether you are attacking something would have helped a lot. Like maybe particles moving towards the target while close to being in attack range or something.
It's an interesting concept though, and it felt pretty nice with the animation when changing color.