Yea perhaps having the ability to have no mask would work better. Or increasing speed of switching to motivate more constant switching. Its hard to say how much that is the issue moreso then the frustration on other parts making it the the scape goat. I dont think limitations on visibility is the main cool focus of the game though I think its moreso going across islands wearing different magical masks. If you were to have the option to have no masks I defiently think you should add more cave areas though where the light mask is needed!
Also if you have time would love to get your opinion on my game! Even if this is your first 3D game you defiently have good taste!