"A 4.5 with 10 ratings is not the same as 4.5 with 20 ratings" - absolutely, and itch will not know what you would think, that could be a 1 or a 5 or something in-between. So in reality if you see a game with only one vote of 4, you cannot make much assumptions, as after you played it and added your vote, it could be anything between 2.5 to 4.5. And that itch has this is good, I'm just saying my change would not make it worse.
Well, maybe a better solution would be to use the median, and add some extra calculation layer on top of that so that not all games have too similar scores - maybe X% of the score comes from median, Y% from top Z%-percentile, Y% from bottom Z%-percentile, etc, etc.
Another change could be that you would need to have downloaded a game, and waited at least 5 minutes after that, to be able to vote it. Probably still wouldn't really turn down trolls though.
And sure, there are good faith 1s. But also note that a 1 will always affect the score (except in my kind of bad and unrealistic all 5's example). But if you see the following examples, the system only makes it so that if the scores are centered high (2 3 4 5 5 5 5 5), your 1 vote affects the score less and if the scores are really low for a game (5 4 3 2 2 2 2 2) your 1 vote actually affects the score more.
"Ohhh, you assumed the ordering of the ratings were of significance. I just counted up to have every different rating, except 1 and added some 5s. Then I added a 1 and then removed a 1 and a 5 to show that the average still goes down if you remove a 1 and a 5 and thus showing that this system does not preserve the average. " Yes, it is intended that the score goes down in both cases. See above, the 1s just have less or more of an impact depending on the vote distribution. The vote still counts. Lets say a game has an average of 4. As it is now you could vote 5 and slightly increase the average, or vote 1 and tank it. I.e. inherently your vote affects the score more if you vote 1. This change just adjusts that.
"But no, you cannot claim any knowledge about justification here either and it also goes again into this assumption of a correct objective rating. Where any nonconforming rating is somehow unjustified." - again, it just weights the value of 1s lower if the distribution is high and higher if the distribution is low. So nonconforming rating are still counted. It is not like your vote doesn't count.