itch.io Community » itch.io » Ideas & Feedback

The rating system doesn't really benefit anyone.

A topic by lionelgold created Feb 01, 2024 Views: 644 Replies: 3

Viewing posts 1 to 2

Hey, sorry if this has been brought up before, and I'm sure it has, probably several hundred times, but I'm at a loss here.

Why are ratings and reviews the way they are? The ability of anyone with an account to rate any game (That has it enabled) regardless of whether they've actually downloaded it, and without providing any sort of context to their rating seems crazy to me.

It makes sense in the real world where google or yelp or whatever can't tell if you've actually eaten at a restaurant, but this isn't the real world. This site definitely knows whether or not you've actually downloaded a game, so why isn't that a basic requirement for actually rating it? Why isn't at least some sort of reasoning for your rating required? If someone who only likes fps games decides to download a puzzle game for some reason, why does their 1 star rating hold just as much weight as the rating from who hasn't just downloaded the wrong thing? Why does their rating hold the same weight as someone who's never even played it?

Obviously this can happen in any industry, but in pretty much every other setting you can see individual reviews and ratings. You can see who rated something and why, so if you know you disagree with that person or their criticisim/praise doesn't align with what you're interested in to you then you can just disregard it. You can't do that here. If you're gonna have a system where you can report a review for having nothing to do with the thing they're actually rating, why allow people to rate something without explaining their reasoning?

Rant out of the way, suggestions.

Similar to how devs can turn ratings on or off, allow them to toggle on or off the requirement to leave an actual review with the rating, with some sort of minimum character count. Obviously this will mean devs who enable this will get less ratings overall, but as long as that's clear the choice would be greatly appreciated.
Allow the dev to make reviews public. All or nothing. If a dev wants to show their reviews, then they show them all, good or bad. At least this would give people some sort of context for why the rating is what it is.

That's it. These two changes. I'm convinced this would solve basically all complaints about the review system immediately, and I challenge anyone to find a single downside to them. Let the overall rating of a game hold the same weight, but let devs choose to recieve what will probably be fewer ratings overall in exchange for more detailed feeback, if they want. I only hate bad reviews if they're not about the actual game, or if there's no reason given. They could be someone who genuinely just really didn't like the game and has perfectly valid reasons for it. Or it could just be someone who just rated it poorly because they were bored and haven't even played it. I have no way of knowing. These changes would immediately solve the problem. I cannot fathom why something like this hasn't already been done. This can't possibly be the first time this has been suggested, right?

Like Reply

redonihunter317 days ago(+1)

Why are ratings and reviews the way they are?

Very likely to save staff from workload for a thing that will not get paid.

The way it is now, I see very few to none cases, where someone tried to fake the ratings. And your example of the player liking fps and trying a puzzle is not valid. For once, that star rating has to have the same weight, or the rating system has no right to exist at all. If you weight the rating of people how would you do that? Have people with a more important opinion?! Why not let the developer rate their own game while we are at it? Should we make qualifying tests to see our expertise in a certain genre?

But the bigger issue is, that is not what I see happening. Most people do not bother to rate. At all. Someone rating each and everything they come across is the exception. And should they do, well, why should that rating be not counted? Sure, there are dishonest ratings, that rate down (or up) things by other criteria than you would apply. Like not liking the genre. Or the developer. Or tools used to create the game. AI comes to mind.

What is a rating worth, if you let only players rate it, that like that content in general? Hmm. Let's think about it. Oh, maybe lot's of seemingly non mainstream things would get sorted high in the highest rating section. That is actually happening, because, as I said, most people do not bother to rate things. So content that is not mainstream will attract their target audience and get high ratings. Just look at the browse section and the highest rated games. Those are not rated by the unbiased general player base.

So in other words, what you want is already happening. Puzzle enthusiasts will rate puzzles. Fps enthusiasts will rate fps games. The once in a while misplaced player that does not like the topics of a game will not distort the result much. One could even argue, if you mis-advertise a game to attract a player that will be disappointed in the game, the negative rating is well earned.

So what about blocking ratings from people that did not download a thing. Tricky. If people want to give dishonest ratings, they will find a way. Maybe it could be implemented for paid games. But you also have to consider cheating in the other direction. Developers giving themselves fake 5 star ratings. And that is easily done for paid games, as the creator can give away the game to fake accounts to get their ratings. But this is likely to get detected and the game to be banned. And if a developer sees lots of 1 star ratings of seemingly fake accounts, that can be reported, should it not get detected.

What I would improve on the current rating system are two things. Show the distribution of ratings. It is interesting to see if your 3 star rating is 2 times 3 or one 1 and one 5. Or if that 4 rating consists of many 4 or mostly 5 and 1.

And making it more clear when rating, that the review is not seen on the game page.

To find games you can try to find collections that have games you like. Either by trawling the feeds or picking one of the collections you see when browsing. You could even follow one of the accounts that regularly rate games to your tastes, so their ratings will show up in your feed.

The general problem for indie games is the relative scale to rate on. 5 stars for Dark Souls is not the same as 5 stars for a hobby game clicked together in a game engine. To me, the biggest information is not the rating score, but how many ratings there are. Because people tend to not rate very much. And mostly in their bubble of favorite topics. So a score of 4.9 for a puzzle game means nothing for a fps player. So puzzle players love the game. Meh. But 700 ratings in an environment where most games have 0 - 9 ratings means a lot, even if the score is 3.4. But also, if you do have those 700 ratings, the few dishonest ratings will not skew the result much.

Like Reply

lionelgold317 days ago(+2)

You're missing my point, probably because I explained it in a fairly over the top emotional way, which is my fault. But how can you tell whether a rating is fake? As a player, you can't even see them, as a dev, you just get given a number. The only way to tell as far as I can see is if the rating is either a word for word copy paste from the same guy given to multiple devs, or if the rating comes from someone whose comment history would imply an interest in something that clashes really hard with what you make. Like if you made something related to LGBT, and you get a 1 star review from a guy whose comment history is 99% him commenting on games involving alt right stuff. Fairly obvious they're not gonna like it, but they can rate it without justification anyway. The whole site is meant to be a platform for indie devs who tend to be on the smaller side, so individual reviews and ratings can matter. I suppose it's probably obvious that this is my non dev account, and my other account is a dev one. No way I'd be up in arms over this if I hadn't seen both sides.

My primary point is that I think people should have to have at least downloaded a game to rate it, and I think that developers should have the option to enforce "review required" when leaving ratings, if they choose. This isn't gonna be a full solution, but it's at least something. This way if someone wants to leave a bad review, they still can, but they have to at least type something. Worst case it turns out to be nonsense (Which could be ignored by the community if reviews were public) and best case it could be good constructive feedback that could help the dev to improve their game (And inform the community about the state of the game if reviews were public.)
Currently you could get a 1 star review and never know why. That's what the point of my example about the fps and puzzle thing was about. It's a fairly ludicrous example, but given how the review system is at the moment can you actually guarantee that it doesn't happen?

It isn't a huge issue, it doesn't even need to be the small issue that it is. The rating menu thing already prompts you for a review, and if devs just had the ability to make it so you're actually forced to leave a review instead of just a rating without any context then there you go, that solves it. Devs who want to take feedback from players can enable it if they want, and devs who like things the way it is right now can just not, and nothing changes. Plus, if players could see reviews, they could read them if they want, or ignore them if they don't.

All of that said, I agree with your point about showing distribution of ratings, and making it clear to players that their reviews aren't viewed by anyone other than the dev and the reviewer's followers. On the rare occasion that I get words in my ratings, they're often written for an audience of some sort, so I get the feeling the few people who do actually leave words in their reviews do actually think someone other than the developer might read them. I do still think that the option to enable "Review required" has no downsides, and whilst it may be difficult to enforce the "download required" idea is also not something I can really see a downside to other than the very specific circumstance of perhaps someone downloading and playing something and then losing access to their account and needing to make a new one, then deciding they wanted to rate the game or something. I just think that actually experiencing a product, if even for a moment, should be required before you're allowed to do your part to affect the overall rating of that thing. I'm no web dev, but these sound like incredibly simple features to add. Sure testing would be required, but still. It even works for the people who write reviews because they want others to read them. I just don't think there is anybody that these features wouldn't benefit, other than the web devs who would have to make them work.

Like Reply

redonihunter317 days ago(+1)

But how can you tell whether a rating is fake? As a player, you can't even see them, as a dev, you just get given a number.

Depends on the cause of the fakeness. If the rating is for the wrong reason, it is indistinguishable and it is debateable, if the rating is even wrong. Just because I do not like puzzles makes my 1 star rating not invalid. But why would I bother to rate or play a puzzle at all. At worst the algorithm now thinks, I like puzzles in general and populates my suggestions with other puzzles.

But fake ratings are usually made with fake accounts. A dev sees who rates. If the devs looks at several incoming ratings and they are from new accounts with 0-1 follows, it is quite clear. One just does not get a buch of 1 star ratings out of nowhere without reason. Either people think your latest update sucked, or you are targetted by fake ratings.

A single fake rating is of course not noticeable. But a single fake rating is not what will cause problems.

Also, if a dev can see the pattern, the itch system can do so as well. And this goes for fake positive ratings as well. There are extensive ratings discussions in and about jams. I think itch knows how the rating system works, and where the flaws are and how to tackle them.

You seem to think that people write comments in their ratings. They do not. Look at https://itch.io/feed?filter=ratings I scrolled quite long to see some comments. Maybe 1 in 50 or less writes a review.

Also, any player has all the right to vote on every game. Otherwise you can forget the rating system completely. If your alt right choses to rate a lgtb game, all the bad for him. He now will have lgbt content in his suggestions. I know, because something similar happened to me. I voted on a game that had a certain tag and suddendly games with that tag were suggested to me, only I do not like that tag. I sat it out and rated different games. Others might chose to 1 star those games. It is not very fair, but you cannot implement any qualification test, if someone is eligble to vote on certain content. Chosing to voice an opinion is a big hurdle, and the proof is the many games that have 0 ratings.

If you suspect a hater, you can report the account. Abusing the rating system is very likely against the tos. So if that person did vote on dozens or hundreds of games, just to deal out 1 stars, they will take action. But if anyone chose to 1 star your game specifically, even if they just did not like your username, that is their right in the current system. And if that person was the only one that could be bothered to vote on your game, maybe your game deserves that 1 star. But if your game is good or even mediocre, that 1 star will not matter. 10 ratings with an average of 5.0 look equally suspicious.

And to add to it, the target audience usually is smart enough to know, that certain content will get hate votes. It is almost expected. I would be suspicous to see a game with AI content, that has 5.0 average, for example.

I think that developers should have the option to enforce "review required" when leaving ratings, if they choose. This isn't gonna be a full solution, but it's at least something.

11111111111111111111111111111111111111111111111111eleven booo stupid stupid.

To paraphrase it, this will happen if you do naive requirements like that. A public review system might be a good idea, but never force the general audience to do things. It will get bad. At the very least, even less people would rate the game. And others might rate it down out of spite for forcing to write something in the rating.

With public reviews you then could do things like Steam or Amazon and show a veryfied purchase. But you do not even need an account to buy on Itch. Or play the free stuff.

They could also do silent stuff in the background and count verified purchases and verified downloads differently. But in the end, why bother. There are just not enough ratings for most games to justify it. And for the games with many votes, it does not matter. And for people abusing the system, those can get cought by the system (and hopefully this is done and they just do not talk about it.)

Currently you could get a 1 star review and never know why.

Well. You also do not know why you would get a 5 star rating. To stay with the lgbt example, maybe someone interested in the topic gave you a sympathy vote and not because your game is any good. That there are haters is known, so someone might have felt obligated "to even the odds". This happens all the time in real world. Virtue signalling is one form of the phenomenon. Rating certain games up another. Rating up indie games for that matter, just because they are indie. It is discrimination. A rating not because of the merit of the game, but because of other stuff.

but given how the review system is at the moment can you actually guarantee that it doesn't happen

My opinion is, that a guarantee is not necessary. If someone abuses the system, that can be dealt with. All other cases are just people being people. And since most do not bother to rate anyways, each vote is in itself a positive thing. Singular outliers are to be expected in any statistical thing. It evens out. Your 1 star rating will be canceled by a sympathy 5 star rating. And the 3 star rating for your run of the mill game that would be accurate is ignored. (That was just for the sake of argument, I have of course no idea if the game would be any good, but chances are, it is an indie game that is not popular/professional enough to be released on Steam, while on the scale of indie hobby games it might be 6/5)

On the rare occasion that I get words in my ratings, they're often written for an audience of some sort, so I get the feeling the few people who do actually leave words in their reviews do actually think someone other than the developer might read them.

People do get confused by this a lot. But actually, there are public ratings and reviews. You just cannot see them from the game page. Visit the global feed and click ratings. Users have to have that enabled (or not disabled) in their settings. Just as the little comment box says, it is for your followers and for the developer. Only most regular users do not have followers ;-)

That requirement for rating power is a nice idea on paper. But you talk about abuse of the system and you can bet, if someone wants to give you a bad rating or someone wants to fake good ratings, they will find a way. It would only work a little bit for paid games and negative ratings.

In the meantime you make the hurdle to give ratings bigger for all and players give very few ratings as it is.

A better implementation of systematic rating abuse detection would be more benefitial, I think.

Any rating system has its flaws. The biggest complaint people on Steam seem to have, is the inability to give a neutral rating. They only have negative or positive. The rating system on Amazon is flawed because they do not fight bought ratings enough. The rating system on eBay is flawed (or was, been some time since I used that site), because big sellers have ways to remove bad ratings.

So my guess is, no matter what would be changed to make it better on Itch, it would open another way for it to be worse.

Like Reply