Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
Tags

Why are some scores lower than the raw score?

A topic by Mr_Men created Aug 16, 2022 Views: 262 Replies: 8
Viewing posts 1 to 4
Submitted

There's a little asterisk (*) beside the score label, but no explanation to what that  it means. Or why some scores are lower than the raw score. Can this be explained?

Submitted (1 edit)

the explanation is given below:  "Score is adjusted from raw score by the median number of ratings per game in the jam."

To expand on graphicmismatch’s answer: the way this works is that if a game has at least as many votes as the median number, then its score is unchanged; else, the score is diminished, using an algorithm that is not public (although empirically, it looks like the farther below from the median a game is, the more penalty it gets).

(Which is an extremely unfair system in my opinion — I think there should rather be a fixed quorum of votes (either in absolute value or percentage) —, but this is another discussion.)

Submitted (1 edit)

This is why I made this post and then scroll down to my reply to  the seventh entry.

https://itch.io/jam/gdevelop-game-jam-2/topic/2290528/can-we-get-to-10000-rating...

Oh wow, I predicted the median of 20 exactly!

As a coincidence, I had seen your thread yesterday. ;)

Actually, your suggestion there yields an illustration of what is wrong with this system: by the very definition of the median, there is always (at least) 50% of the games that will receive at most the median number of votes. Therefore, it is entirely possible that half of the games get a downgraded mark, even if they were reasonably reviewed.

The only possible situation such that no game receives any penalty is that the 50% least rated games all get the very same number of votes. This is very specific and does not make sense in my opinion.

I have no idea whether itch.io is considering amending the system (which I think they should); you would have to look (in their forum?) if they address this issue.

Submitted

Yes, haha, I was aware that the motivation for my suggestion may have been flawed, this is the kind of thing my brain doesn't cope well with.. You say it's possible that half the games would be downgraded but I was wondering whether this would be  a certainty just from the nature of a median value. 

But it does make sense to have some kind of downgrade system so that a game with one five star vote doesn't win. For me, the solution lies not in the results system but in the voting system.  After playing one game that must have been entered into another jam as well, I must have clicked on the wrong entry page link as it was telling me I couldn't vote for the game until I'd voted for some others with few votes. I feel like forced voting like that could lead to just voting a low score without even playing just to get back to rating other games faster.

I think I saw some mention somewhere of some jams where  the submissions only have one ordering  filter, that is the games that have voted the most are displayed at the top. But once again that could lead to dummy voting.

I can't really think of a good solution that doesn't rely on people doing the right thing. I think the biggest contributor in this jam to the ratings gap was the rate for rate posts. I don't know if all jams are 'plagued' by such posts but I can't see any way of having a fair jam while they exist.

This is an interesting discussion. :)

Yes, haha, I was aware that the motivation for my suggestion may have been flawed, this is the kind of thing my brain doesn't cope well with.. You say it's possible that half the games would be downgraded but I was wondering whether this would be  a certainty just from the nature of a median value.

No worries. ;) The median is a value that separates a statistical sample into two same-sized groups. Notice that this means there can be several values equal to the median; for example, in the data set 1—5—5—5—7—8, the median is 5 (with subgroups 1—5—5 and 5—7—8), but only 1 is strictly below 5. I expect that, most of the time, in a game jam, there will be just a bit below 50% of games that get strictly below the median, and thus get a penalty. In some fictional example (which is actually close to a real one I know, where a game could have ranked above mine by one place!), I think it makes about zero sense to downgrade a 15-vote entry but not a 16-vote entry, changing their ranks in the process, just because the median is 16.

But it does make sense to have some kind of downgrade system so that a game with one five star vote doesn't win.

Well, I was thinking you can either have a downgrade system, or simply some kind of indication or ranking without entries with too few votes (and one with all), but as I said, I think this should rely on a fixed amount of votes to get. This may not even need to be proportional when the global number of votes is big enough, because I think I remember seeing that a representative sample does not need to be proportional, merely big enough in absolute terms — still, double-check this. ;)

For me, the solution lies not in the results system but in the voting system.

I agree that the voting process has problems too, and could be taken care of. Actually, I would like both the results system and the voting system to be reformed.

After playing one game that must have been entered into another jam as well, I must have clicked on the wrong entry page link as it was telling me I couldn't vote for the game until I'd voted for some others with few votes. I feel like forced voting like that could lead to just voting a low score without even playing just to get back to rating other games faster.

I think what you are referring to is the rating queue. You may be right that this leads to such a bias, it occurred to me too. I think this is even more serious in game jams where public voting is enabled (and public voters are often forced to go through the rating queue, while not necessarily the submitters): I guess this tends to bring in fickle voters that do not hesitate on being overly harsh. The rating queue is an alleged solution, but I am not convinced (for example, I have seen several worthy games from GMTK Game Jam 2022 with both few votes and weirdly low raw score).

I can't really think of a good solution that doesn't rely on people doing the right thing. I think the biggest contributor in this jam to the ratings gap was the rate for rate posts.

I am thinking of at least two things:

  1. Game jam hosts should make it more obvious in the main game jam page and forum to use special orderings (especially the link to the special list of low-vote entries that pops up at the top of the submissions page), explaining how this is important and removes the need for rate-for-rate threads (and even forbid such threads when the reform is in effect);
  2. The Karma ordering could be more hightlighted. In case you or anyone else wonders what this is: this ordering originates from Ludum Dare and considers (using some formula) the ratio of the votes you cast (and number of others’ pages where you comment, I think) over the votes you receive; this way, someone who volunteers to review other people’s contributions gets more spotlight in this ordering, until he gets enough votes to balance it. This is somewhat akin to a more systematic and moral version of the vote-for-vote system: the more you review, the more your game is reviewed. (Nevertheless, I read in some comment that the system (either here or in Ludum Dare or even both) had experienced changes and did not work as it used to, but I am not aware of the specifics.)

I would even consider removing the popularity ordering during the voting period, I feel it tends to create a vicious circle, seeing the discrepancy in how many votes each game can get. It is just my intuition, though.

A more complex suggestion would be to enhance the rating queue system by asking the voters to write a reasonably detailed and fair review, in order to avoid careless marks. This system may be hard to check…

I don't know if all jams are 'plagued' by such posts

It is apparently quite common (at least in large enough game jams, I think); I have seen this ostensibly in Mini Jam, it happened here for GDevelop Game Jam #2 as you said, I have seen it for GMTK Game Jam 2022 (including on the Discord server), and a message from this thread on the Karma system seems to confirm (‘The current reliance on spamming the community with "Play my game" and "I'll rate you if you rate me" in order to achieve an above mean average rating count is frustrating and discouraging.’). On the other hand, I did not see it in a smaller game jam (Gamecodeur Game Jam #36), although I did not check the Discord server.

TL;DR:

  • Ask for a fixed minimal number of votes;
  • Make the least-voted and Karma orderings very obvious both in game jam main pages and on the site.
Submitted

I didn't know what the karma ordering meant and read the link. I see the itch boss said it was given - received with no mention of comments but that he may tweak it.

Thank you for showing that median can be more complex than a 50/50 split. It would be interesting if the jam results had the number of votes as a sort field.

Interesting topic indeed and still not solved in my book, haha.

If there were a fixed vote requirement then just like now with the median there's always going to be those who miss out by one vote and feel that life is unfair :/ (don't know how to visually type a protruding lower lip).

I'm not sure how effective it is to make it clearer about games in need of votes. I felt that the placement of that message in this jam was pretty good. One factor is just the number of entries. It's just not feasible for most people to play and rate every game. Which then leads back to the same problem, people know they can't rate all the games so they go for the option where they can get something in return - rate for rate. And I'm sure there's larger jams than this one which means being even more selective with one's time, either by rate for rate or by playing what is deemed to be popular in the hope that it will be good.

So, my summary is the more entries a jam has, the less chance it has of having a reasonably even spread of votes.

(Passing by after being massively busy!!)

:/ (don't know how to visually type a protruding lower lip).

I do not know either but there are some webpages out there with compilations of emoticons, you can surely find this! X)

So, my summary is the more entries a jam has, the less chance it has of having a reasonably even spread of votes.

Ludum Dare is a rather big game jam, and you can easily get the voting quorum (which is about 20, so, a fixed number) if you review other games. For example, I have taken a look at the last edition (which is 51), and those people that do not reach the threshold are those that reviewed few games, the others generally experience little to no trouble. (Except that the new algorithm still needs some fixing.)

Notice it does both use a fixed threshold and a karma-based default ordering, which hints at it working.