I've been closely watching the feedback people have been giving about the results of the jam. There's a lot of discussion going on!
I thought it might be interesting to create a JSON export of all the results so people can experiment with coming up with their own ranking formulas (or do any other analysis they want). I can't say if the outcome of this topic will impact the results of the jam. I'm not the host of the jam, just an itch.io admin. But, I think this could be cool way to explore how itch.io may handle ranking results in jams for the future.
Before we get to the results file though, I want to explain how the current ranking system works.
Current ranking system
The score formula:
final_score = average_score * sqrt(min(1, num_ratings / median_num_ratings))
- average_score is the average score of all the ratings you got
- num_ratings is the number of ratings you got
- median_num_ratings is the median ratings per submission across all submissions in the jam
In plain English: For submissions that have number of ratings ≥ median, the average score (aka raw score) becomes the final score. Submissions that have less than the median number of ratings will have their score penalized proportional to how few ratings they got. The sqrt is used to make the penalty falloff a bit more gradual.
Approximately half of all games will have a penalized score (due to the use of the median). The idea is to balance between not giving popular games with a lot of ratings an advantage, and letting lesser known submissions have a chance at having a high score. The median puts that balance right in the middle.
A submission's rank is then calculated by sorting all the submissions by their final_score
(It's a large JSON file)
Looking forward to seeing what people come up with
Note about ratings
This jam uses Public Voting. This means that anyone with an itch.io account can rate a submission. Although this gets a lot of people to participate, I personally this think method is inherently flawed. It can encourage cheating, and it often ends up detecting who has the largest network of friends available to vote on their project. (The alternate rating method has only people who have submitted to the jam have a vote). Just something to consider.