The enemies can move and attack before you can, that makes the hard gameplay (which is fun when you figure it out) even harder and includes something that can't be overcome with skill or understanding of the game. It makes it unfair. You can't put in a mechanic that kills the player before they have control of the character.
Other than that, solid foundation but it is indeed incredibly hard, especially the beginning. I'd include different difficulties and maybe a tutorial level. Also the beginning needs to be easier. You could for example make the yellow one slower and then speed it up at 5 or 10%. Like how it is right now it's incredibly frustrating, but I like that you don't make players start from the very beginning when they die at level 2.