Well, for my understanding, "losing a combo" isn't really the problem, just make that missing the beat interrupts the combo, like any other rhythm game. The thing is that whitout obstacleas you don't really have a reason to switch side/direction. Then you're only left with the difficulty problem, that i think it can be solved by adding one or more super slow chill track
Modular difficulty also sounds cool!! Too bad that you didn't had time to implement it :/