Concept is really cool you have good ideas.
I'm stuck at level "Red is reset". Sadly a bit too hard for me. ^^'
My 2 cents : I think you need to choose if you want a frenetic puzzle game (with short limited time) or dexterity puzzle game. Both seems complicated. Same for the transitions.
- If you want a die and retry, your level transitions are a bit too slow. Instead of big black screen with text, you can add a footer text like VVVVVV.
- If you want a dexterity puzzle game, you can enlarge timers or remove them to let the players experiment with your environment. You can even let them choose when to skip to another character with a right click button.
About the controls, I don't think the mouse is the real problem. Perhaps you can improve your gamefeel by using another reference frame for your movement offsets (I don't which one yet but with some little experiments you can find something).
Despite all of this, I think you did a really good work and your game looks really promising.
Good job! :)