Thanks for the feedback!
I already had some draft version of a difficulty setting that player can select but it was finicky (especially when player changed it during the actual game) so I left it for later. From your comments it seems I might have to comeback to it earlier than I thought. Maybe I will let player select the difficulty at the start and thats it.