I dig this idea, and I think the difficulty is kind of fun, with some polish could make for a solid little roguelike or something