This is very creative! I thought of going with something similar: parallel realities that are switched as you play, but that would require too much art effort; you were smarter here hehe. There could be some levels instead of a continuous game, though, to kinda checkpoint it, but I get it that restarting from zero is part of the challenge (I created a roguelike so I can't complain :)