As I chose the tag "idea" it is my understanding that it is okay to propose ideas for improvements here.
Please make the threshold for penalising jam entries modifiable by the jam organiser. The median can still be default and be your recommendation. It isn't a mathematical truth that the median is always the best decision. The organiser might be having the same jam year after year and has a better idea of how many games and rating there are, and why some games in a community get less ratings.
I think I mentioned this in another thread, but the goal is not to avoid giving a penalty to projects, but the goal is to come up with fair way to relatively rank every single project against every single other one. By factoring the amount of ratings a project has will avoid any “unfair” moments where a project with low number of ratings with less statistical significance ranks higher than a project with a similar score that has higher number of ratings with higher statistical significance.
That said though, in the future we may expose this option for hosts but given the complexity of understanding the dynamics of the ranking, by having this as an option, I’m concerned hosts may choose something that they may think is better but actually reduces the quality of the relative ranking.
Are you hosting a jam where you think that the median has degraded the ranking?
Hi leafo, thank you for answering. I think most organisers want to be as fair as possible and that it can be tricky as there is a lot we don't know and people might cheat etc. I agree that we don't want a game with e.g. two ratings of 5 to win a competition. On the other hand, it is not necessarily fair to assume that all games with few ratings are overrated. They might as well be underrated.
I am from the interactive fiction (IF) community (text adventures) and I participate and/or vote for games in the annual TALP Jam and ParserComp and Ectocomp plus several non-annual competitions here on itch and the subject has come up before. One hurdle is that some games are accessible on more platforms than others, e.g. not all are playable in browsers etc. so in practice, some platforms are significantly penalised. IF games usually require interpreters which people in the community may help developing so already written games become accessible on more platforms. If new, promising platforms are always significantly penalised, they will continue to rank low and people won't give them the attention they deserve.
