That's true, some of the early stat check can only be passed with your starting attribute. Some also provide you with an option of losing points with the girls but increasing your stats (by learning not to do it again).
In the long run, failing stat check won't matter much. When there is an important stat check that you need to pass, it'll be included in the quest requirement.