From my view, you propose a solution to a non-issue. And to make it worse, I do not think your solution would work.
The solution to rating abuse is to remove the abusive ratings.
So, would your solution make it any better? The harrasment ratings would still be given. The developer would still be harrassed. The score would still go down, no matter how you calculate the average. And if the issue is already "fixed", why bother with removing fake ratings. We have implemented a solution after all.
So, would your solution make the rating system better for everyone? Like developers with very few ratings? Or users expecting the rating score to say what exactly? An important point. What do users expect the rating score to say? Do they expect a simple average or do they expect some convoluted math that flattens the rating and makes it nice?
On Itch, giving a rating is already a rating in itself. 1.0 million of the 1.3 million game here do not have a rating score. My advise to people trying to downrate the thing they do not like, is meant serious, even if it might read tongue in cheek. The smirk between the lines stems from the irony that people downrating things are actually promoting the thing they do not like. But again, if they used fake accounts to manipulate ratings, that should be reported. Each rating has a report button, in case you did not know. But that should only be used for fake and abuse and harrasment and not to try cull 1 star ratings.