There's no downside in having multiple difficulties. If you have a certain difficulty that you feel the game is at its best, just mention it( "e.g 'Hard' - 'recommended for the authentic experience' or something), or make it unlock-able so players can new game+.
Difficulty on the fly as an option is pretty nice too, because some games can be decently difficult through most of the play-through, but have one moment where everything spiked to eleven and you have no idea what to do.
Another way to tackle difficulty is to give the players tools that could make the game easier to play at the expense of their high-score or something like that.
(Not too sure how to explain but for example you'd give a long range weapon in a very Close Quarter Combat focused game)