Skip to main content

Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
TagsGame Engines

Dissatisfaction with the rating system

A topic by NikX04 created 20 days ago Views: 277 Replies: 8
Viewing posts 1 to 8
Submitted(+3)

I want to preface this by saying that the result of this Jam is not that important. The most important thing in all game jams is getting to meet new people, learning new skills and developing teamwork/management skills and gamedev skills. 

The current rating system does not work well. For a public rating system (one without judges or jury) for a competition like this to work properly, it must ensure a fair rating count for all entries. That is because you cannot properly directly compare the average rating of two different entries which have had different review counts.

Problem 1 - Rating count gaps
Which of the two games should win?
A game with 4 reviews - all coming from a rigged Rate For Rate post - all of which are full stars in all categories is compared to a game with 100 reviews with varying reviews. On average the game with 4 reviews should straight up win the competition, since it has a perfect score. The other side - A game with 100 reviews might win by review count alone if the system is made in such a way to favor rating count instead of rating averages. There are many ways in which the winner might be calculated to compensate for review count gaps, however, the wider the gap between individual game review counts, the more unobjective and randomized the result is.

For a more fair rating system, it would be beneficial that all games get around the same amount of reviews. Now, that does not mean that each game should get 10 reviews for example and not a single more - it should just mean that there don't exist games with 2 reviews, while there are other games which have 200 reviews. The more the system has to compensate for rating count differences, the worse the system will be overall.

Problem 2 - Downloadable VS Mobile VS Web
The less clicks until game start, the more plays and ratings
Take a look at this post: A Very Long Rate for Rate post! PLEASE READ - Brackeys Game Jam 2025.2 community - itch.io
TL;DR; Nobody is playing or rating downloadable PC games. Mobile games have it even worse. Everyone is playing web games.

This is a massive issue. Web does not offer as many features as PC or Mobile. Both PC and Mobile games can be much richer in terms of game design than web. Web basically forbids all 3D game concepts, except the absolute simplest ones. Web games mean - No DirectX, No Vulkan. Please take a look at the steam store and even the itch io frontpage. How many killer games have been made for web in the recent years? 1? 2? Friday night Funkin is literally the only game I could think of that could rival standalone indie PC and mobile games in recent years.

Problem 3 - The Jam search filter system
The search filter system should be gone. Straight up, random only. It is so easy to sort by "Most Popular" and then boost the games which are already at the top. There are so many great games in this jam which I have personally favorited, which have like 7 reviews and 2 comments. Most Popular is a slippery slope. If you just get there once at the start, you will gain momentum and snowball into even more reviews. This further aggravates our first problem and serves to create even bigger gaps in the review counts of games.

Problem 4 - Rate for rates
This issue is self explanatory. It is just straight up competition rigging. Want to get your game seen? Just spam R4R posts every single day of the jam. The competition should reward game design skills, programming skills, art skills, sound design skills and teamworking skills. The rating categories reflect this, however, rate for rate posts only serve to reward people who spam the community tab. 

Problem 5 - Gray area win tactics
There are random ways in which the system can be played, which are not against the rules, but are obvious crutches that only serve to boost engagement for games that may or may not deserve it. There are plenty of these, but one of the most obvious ones is the following:
Game Jam starts -> Immediately publish an empty project (literally nothing) -> Your game is now on the TOP of the  "Oldest first" Filter

Possible solutions to these issues:
The insane solution:
The only way to play games is to have them served to players by an algorithm. No filters, no searchbox. "Play random game" as an only option in the jam menu. When that button is pressed, an algorithm choses the game with least current votes and serves it to the player. The player cannot play any other games until the served game is rated. This continues indefinitely. This will ensure that all games have the same review counts, will kill all R4R posts, will kill problems with advertisements and will also get rid of the "Nobody is playing downloaded games" problem.
I am aware that this is NOT going to happen.

The normal solutions:
Problems 1 & 3 - Remove the filtering system (permanently switch it to random), temporarily hide games with huge rating count leads.
Problem 2 - Boost downloadable games chances to appear.
Problem 4 - Ban R4R posts.
Problem 5 - Ban and prevent any activities which might serve to "cheat the system". This should be a competition of skill, not one of "Who can abuse the website mechanics the most". If you're on top of the "Oldest first", you should be there because you work thrice as fast as the competition and have actually finished first.

Don't take this the wrong way! I am endlessly thankful to itch and Brackeys for creating this game jam, it was a great opportunity for me to meet new people and expand my game dev skills. In my mind we have all already won by creating great games and expanding our skillsets. It would just also be really cool if the entire "competition" side of this was a bit more well thought out.

As we all get feedback for the games we make on here, which helps us grow, it is only fair to give some feedback to the ones who gave us this great opportunity in the first place as well.

Have a good one! (And please go download some games, there are Untitled Goose Games and Lethal Companies with like 4 ratings in this Jam lol)

Submitted (1 edit)

Hello, I think this is a great post, personally for this jam I managed to do amazing however in prior I did horribly. I think all you points are extremely thought out and I agree with most if not all of them. With rates for rates, I personally don’t think there nice, but I think “Do you want feedback?” pages are completely fine, even if it gives the person making it a burst of views and ratings.

Thanks for sharing thing!

Submitted

You make some good points. One thing I’d add is a weighted category system, because surely enjoyment should matter more than anything else. Honestly, I’d probably change the whole system, but a weighted approach would already be a big improvement.

Another idea for supporting downloadable games could be if Itch added a way to stream them directly on the platform. I know that’s a huge ask, but it would be incredible. And if Itch never does it, maybe they'll be replaced by a platform that will.

Submitted(+1)

Second game jam for me.  First solo.  I was amazed by how hard it was to get people playing my downloadable game.  The middle of last week I was sitting at around 10 ratings and was visiting web based games that were already in the triple digits.  

I don't see R4R posts quite the same way as you.  From my perspective, it's about the only way people can market their game on here.  In fact, that's the only way I have been able to get any ratings at all.  I have not posted an R4R thread, but I have been replying to them all day, every day for the past six days and have only got a whopping 32 ratings for my effort.  That said, I have been responding primarily to others with downloadable games that are in the same boat as myself because they are the only ones willing to play mine.  At this time, I have rated 67 games.  That's about two ratings given for each one received.  I feel confident that I would have zero ratings if not for these threads.

Maybe some of these issues can be resolved by capping the number of ratings you can get?  Once you hit that arbitrary number, your game is hidden or otherwise buried so others can get in the spotlight.  Will that solve the problem?  By itself, probably not.  

In the other jam I was a part of, you were assigned a queue of ten games that you had to play through before you could try anything else.  That could help as long as your game gets on enough queues.  I would also suggest that the number of games you have to play would be based on the number of submissions.  

Maybe the jam needs to be split into categories?  Web, downloadable, and mobile and you compete/rate in the category that you submit.

I understand your frustration.  I'm going to head off to play Hellbakers.  Feel free to play and rate my downloadable game.  ;)

Submitted

I'm happy to having found a post like this in the middle of the R4R storm.
I share most of what you said, but I'm not very convinced about the solutions.

Premise: this is my first game jam so my opinion may not be the most comprehensive one!

I think that what is missing here is that a game jam doesn't end when you submit a game and you are not really helping someone by simply saying "nice game" or stuff like that. Unfortunately there isn't much real human interaction and what makes it worse is that those who have the most rates have the least need to do that, but they are missing the best part! Personally I have played all the games of the people who have commented mine and I look for games alternately filtering between Most Karma, Least Rated and Random but choosing who actually participates in the game jam (comments are very indicative for that).

In a real-life context, you play my game, I play yours, and then a dialogue begins. That's how it should work online too.
I don't know how such a result can be achieved, perhaps simply through people having a greater desire to learn and less desire to stand out.

Submitted

Very true that last thing men

Submitted

I agree I hope the organizers of the game see this so they know how scuffed their rating system is 

Submitted

This's my first jam, that feels success the most, yet the point you made were absolutely true.

I kinda think that forcing to vote 5 random games a day before rating other game would do the job (like in this year gmtk system)

Because the only way you gain players in this jam is to be active in community posts fr - 3-

Submitted

i agree on most points except for r4r, r4r is a filter to rate games of people who are actively rating. I have rated twice the amount of people than i have rating myself, half of the people are just not rating anyone or very little. r4r filters active people who actually rate and care for each other.
r4r does not guarantees 5 stars will be exchanged, no way to check it, r4r guarantees you will get rated
i was avoiding downloadable games myself, but after creating r4r thread i am actively downloading them (to keep my promise to rate and review)
so you kinda figthing your allies rn.