Ok turns out changing difficulty in Quick Groove mode also effects Story Mode. Would be nice to have something to tell you that.
