a piece of advice I see in all "game jam advice" videos is to err on the side of too easy, so I wouldn't worry about that, as long as it's still engaging. And yeah, even though it's too easy, a moment of lost focus still can lose you the game. So I feel the difficulty works.
And yeah. Jam time limits do mean sacrifices so I get that... if you work on a post jam release and sync up inputs to the music I'd love to revisit the game.