I understand that this rating formula exists to avoid the case where a submission with 2× 5 star ratings beats a game with 200× good ratings and this definitely makes sense for bigger game jams I’ve participated in with hundreds of submissions and thousands of ratings (like Game Off 558 entries, avg ratings 13.1, median 9). This is not our case.

I also read this interesting post https://itch.io/post/3962733 from a thread started by entrants complaining why they got lower scores, I’m grateful to be able to see what the formula is and understand the reasoning behind it. I also need to highlight this part of the post:

The average number of ratings is always larger than the median

This is probably often true but not always.

I’m organising a game jam this year for the second time. It’s for a smaller community compared to other jams I know like Game Off that gets hundreds of entries. 346 ratings were given to 23 entries (100.0%), the average number of ratings per game was 15.0 and the median was 16. Notably, the median is higher than the average.

Our entries are mostly submitted by people from within our own community, we talk with each other during the jam and we do our best to review all of the games. Last year’s final results were ordered by itch’s weighted ranking and compared the average score (which was displayed next to it) you could see that it produced (what we perceive as) some pretty unfair results. In many cases the Score and Raw Score are identical. However in some cases a game with an otherwise good score, but with slightly fewer ratings, dropped several places, e.g. from 2nd place to 5th or dropping out of the top 10. After comparing the results weighted and unweighted we agreed (including the first place winners of the jam) that we would have preferred it if the games had been ranked by the “raw scores” instead of the weighted scores, as that would’ve been more fair, better reflecting the actual ratings the games got.

I’ve searched for ways to disable this weighting in itch.io and found none, and as a jam organiser that’s a bit frustrating. Hopefully an admin can help us find a way to turn this off for our jam so that we can use the raw scores instead.

tl;dr: For the jam I’m organising our community would like to use the raw average rank instead of the weighted rank for the final scores, please help us set this? The jam: https://itch.io/jam/ebitengine-game-jam-2023

Unweighted ranking by itself is a bad idea, I’ve looked at many jams at this point I’m a strong believer in the algorithm we use. Without weighting then something is needed to compensate.

itchio does provide manually ranked criteria, so if you have specific needs for the ordering of results in a jam you host you can set the final ranking yourself. I do not want to add an option for hosts to use a fundamentally flawed sorting method since it will cause more issues than it would solve.

In the future I may spend more time exploring other algorithms, but please trust me when I say that coming up with a formula that works well in a large range of scenarios is not a trivial problem. The most minor changes will have substantial consequences in scenarios that are hard to comprehend when just casually reviewing the results of a single jam. For those that have issues with the current algorithm, they generally always highlight an issue with a pair of games in the overall ranking. It is very hard to adjust the algorithm in a way that would specifically solve the problem they highlighted having substantial consequences on the rest of the ordering. That is why I consider requests like that as someone just wanting to specify the ordering themselves. So, I recommend doing that by using a manual criteria.

Also, keep in mind that the “weighted score” is not how people perceive your game: it solely exists for sorting. Another gut reaction I see is that someone who may get a 5.0 average on category (eg. Graphics) gets a 3.0 weighted ranking feeling upset. That doesn’t mean their graphics are bad. Their graphics may very well be a 5.0, but there are other games that have higher confidence level for the purpose of ordering.