Skip to main content

Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
TagsGame Engines
(+1)

Sorry for all the text. I got carried away and I can type fast. If you want to see how Itch handles ratings adjustments, there are some discussions about how ratings in jams are handled. There are systems in place to deal with the number of ratings. A 4.5 with 10 ratings is not the same as 4.5 with 20 ratings. There they do have solutions for small amount of ratings. But only to compare them to other games in a similar situation, like being in the same jam.

But there are not really any "good faith 1s"

What does 5 or 1 (or 2, 3, 4) even mean?

On Steam you have a clear question: would you recommend the game? Yes/No? There is no middle ground. Does it mean that a "No" translates to a "1-Star out of 5 - would not play again" rating? Unfortunately, in some way, yes, it does.

Do you rate 1 on games you do not like (for whatever reasons, which includes disliking the company for including a root kit drm system or other meta things, like their dlc policy), do you rate them 2, or 3, or not at all? Everyone will have a different answer here.

Or do you rate 4 on the games you like and only 5 on your favorite games? Wait, there are favorites and there are all time favorites. So better rate 3 on good games, 4 on your favorites and 5 on the things you play again and again. That leaves 2 for the average games and 1 for games you did not like.

I hope you get my point here. Because I very strongly disagree with your notion that there are no good faith 1 star ratings. Players are not game critics that are paid to give consistent ratings of games according to some defined criteria. Everyone has their own valid views on what the 5 different numbers mean and when to use them. And some people will only give 1 or 5 stars. While others will only give 5. People giving the full spectrum are rare in my estimate. Giving a rating at all is already a big deal on Itch.

and when there are, then the average rating would we very close to 2 anyway.

So your assumption seems to be, that a game has an objective rating and players rating that game will only deviate from that rating by very little. Like the objective rating of a game is 2.5, so with the standard deviation of 1.5, people can rate it from 1-4. Whereas this other game has an objective rating of 4.5 and obviously players rating it less than 3 deviated too much and therefore gave a rating in bad faith.

Your solution to just weed out the bad faith ratings and try to preserve the apparant average when doing so, is very similar.

But it is the other way round. The "objective" rating emerges after many (>100 at least) people give their subjective ratings. And this includes people that will downvote a game for lame reasons. Or upvote it for reasons that are not any better.

2 3 4 5 5 5 5 5, adding a one here is not deserved and can be seen as in bad faith

You do not know why the ratings are that way. So you cannot claim any knowledge at all about how deserved a new rating would be. Even if you disagree with the newest rating of 1, it would just be your subjective opinion. People have different opinions. One person's trash is another person's favorite game and vice versa. For all you know the ratings of 5 might have been undeserved.

Giving a rating of 1 is not the defining attribute of a bad actor.

2 3 4 5 5 5 5 5, making it 5 4 3 2 2 2 2 2. In this case a 1 could be justified.

Ohhh, you assumed the ordering of the ratings were of significance. I just counted up to have every different rating, except 1 and added some 5s. Then I added a 1 and then removed a 1 and a 5 to show that the average still goes down if you remove a 1 and a 5 and thus showing that this system does not preserve the average. 

But no, you cannot claim any knowledge about justification here either and it also goes again into this assumption of a correct objective rating. Where any nonconforming rating is somehow unjustified.

The average is calculated by looking at the data. You do not look at the average to say some data is in bad faith.

that gets together to downvote games

That's an abuse of the system. A review bombing. It should be reported to the platform, so they can take action. Itch does take action, but they remove fake ratings in bulk and that can take a good while.

Also, if the accounts are not fake, but genuine persons, they might rule, that this is no different from a book club reading a book and collectively voicing their opinion about that book. Though if they do that systematically to just downvote certain things, Itch might still take action. They also do take action against fake upvoting.

But you were talking singular "bad faith" ratings and how to remove them from cacluation of an average. I shall repeat my opinion about this. A solution for games with only 10 ratings ain't gonna happen. And games with 100 ratings don't need it. And actual abuse of the rating system should be reported to Itch. Giving a 1 star to a game is not abuse of the system. Giving it to hundreds of games, just because they share a tag you do not like, would be abuse of the system though, imho.

To use your example of the 1 star with the gay comment. Maybe the rater would have wanted 2 persons to be gay. Can't be gay by yourself, so the 1 person was seen as a token character and received very negative. Or the person did not read description and was surprised to encounter the topic and was disgruntled by that and left a 1 star. I assume you paraphrased examples. For an actual bad faith example, it is not very bad faith. Bad actors usually would hurl insults and worse in harassment ratings. Or use fake accounts. For both the solution is to report them, and not change the rating system itself.

(3 edits)

"A 4.5 with 10 ratings is not the same as 4.5 with 20 ratings" - absolutely, and itch will not know what you would think, that could be a 1 or a 5 or something in-between. So in reality if you see a game with only one vote of 4, you cannot make much assumptions, as after you played it and added your vote, it could be anything between 2.5 to 4.5. And that itch has this is good, I'm just saying my change would not make it worse.

Well, maybe a better solution would be to use the median, and add some extra calculation layer on top of that so that not all games have too similar scores - maybe X% of the score comes from median, Y% from top Z%-percentile, Y% from bottom Z%-percentile, etc, etc.

Another change could be that you would need to have downloaded a game, and waited at least 5 minutes after that, to be able to vote it. Probably still wouldn't really turn down trolls though.

And sure, there are good faith 1s. But also note that a 1 will always affect the score (except in my kind of bad and unrealistic all 5's example). But if you see the following examples, the system only makes it so that if the scores are centered high (2 3 4 5 5 5 5 5), your 1 vote affects the score less and if the scores are really low for a game (5 4 3 2 2 2 2 2) your 1 vote actually affects the score more.

"Ohhh, you assumed the ordering of the ratings were of significance. I just counted up to have every different rating, except 1 and added some 5s. Then I added a 1 and then removed a 1 and a 5 to show that the average still goes down if you remove a 1 and a 5 and thus showing that this system does not preserve the average. " Yes, it is intended that the score goes down in both cases. See above, the 1s just have less or more of an impact depending on the vote distribution. The vote still counts. Lets say a game has an average of 4. As it is now you could vote 5 and slightly increase the average, or vote 1 and tank it. I.e. inherently your vote affects the score more if you vote 1. This change just adjusts that.

"But no, you cannot claim any knowledge about justification here either and it also goes again into this assumption of a correct objective rating. Where any nonconforming rating is somehow unjustified." - again, it just weights the value of 1s lower if the distribution is high and higher if the distribution is low. So nonconforming rating are still counted. It is not like your vote doesn't count.

From my view, you propose a solution to a non-issue. And to make it worse, I do not think your solution would work.

The solution to rating abuse is to remove the abusive ratings.

So, would your solution make it any better? The harrasment ratings would still be given. The developer would still be harrassed. The score would still go down, no matter how you calculate the average. And if the issue is already "fixed", why bother with removing fake ratings. We have implemented a solution after all.

So, would your solution make the rating system better for everyone? Like developers with very few ratings? Or users expecting the rating score to say what exactly? An important point. What do users expect the rating score to say? Do they expect a simple average or do they expect some convoluted math that flattens the rating and makes it nice?

On Itch, giving a rating is already a rating in itself. 1.0 million of the 1.3 million game here do not have a rating score. My advise to people trying to downrate the thing they do not like, is meant serious, even if it might read tongue in cheek. The smirk between the lines stems from the irony that people downrating things are actually promoting the thing they do not like. But again, if they used fake accounts to manipulate ratings, that should be reported. Each rating has a report button, in case you did not know. But that should only be used for fake and abuse and harrasment and not to try cull 1 star ratings.