Are you pressing the notes when they're on the colored indicators? Some people get confused by the hearts at the bottom of the lanes, but those are just decoration and not the actual hitboxes. Pressing the notes at the wrong time will make you lose faster, so I assume this may be what's happening :O