Okay so, I'm a stubborn person. Went for some dinner but kept having this annoying feeling for giving up. So yeah, came back and managed to beat the dreadful third level and the last one too as that was just piece of cake in comparison to all others.
I think the mechanics could have been introduced a bit more clearly maybe by more incremental level design. Now the game threw pretty much everything at you on the first level and nothing was really clear by itself on what it did if you weren't fully paying attention to the whole level on what changed. I had basically hard locked twice and played 75% of the game before I actually understood everything. Obviously even slight tutorial messages would have helped a ton. Or even having more clear activation animations/effects/sounds drawing the eye to the spot that changed due to button press etc.