Rules as written, you lose it forever indeed. Thematically, it reduces chances of "beginner's luck" kind of success and there's a couple of other statistical considerations behind it. However, I can totally see myself granting an item or a bonus that allows to regain a "lower" die one time.