Skip to main content

On Sale: GamesAssetsToolsTabletopComics
Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
TagsGame Engines

About the new random queue rating system... (I think it is bad)

A topic by barnanemes created 9 days ago Views: 314 Replies: 16
Viewing posts 1 to 9
Submitted (1 edit) (+5)

I have mixed feelings about the new rating system, and I am not sure if it is the best way to go. Just to give this some context I will ellaborate below, but to preface everything I liked the old system much better.

So our team consists of some enthusiastic programmer and art guys, and we do game jams mostly for fun. Usually the whole reason of doing a jam is the thrill of making something fast, and trying to come up with a scope that is actually doable in a short time (and of course fail at that miserably).

But half of the fun is doing the whole community thing after, where we take a look at different games that got popular or very rated and see what makes it so. Until now we kinda did it as a continuation of the game jam, that when the submission deadline was over we went to the submissions and started rating and commenting on games that stood out for whatever reason, or got our attention with their capsule art. And it had a nice back and forth effect where we could comment for comment and rate for rate. I know that having many ratings does not necessarily means that we will have a good score, but still getting comments and getting rated was a number go up game that motivated us to engage more and more with other games and the jams community during the rating window.

I know this should not matter, but I feel kinda cheated with the new system, that I cannot rate whatever I want to. I get the problem that sometimes games don't get enough rates etc, so there should be an egalizing force that helps these entries. But for me having a random rating queue is just not justifiable by any means. I would understand if I could only rate 5 games first that have not been rated / played by a certain number of people, but after a threshold of like 5 or maximum 10 games I should be able to rate whatever I want. 

The main reason for this I think is that playing jam games, and taking them seriously and giving constructive feedback is a lot of time and energy but if I cannot choose what games I spend my time on it becomes kind of a chore. Realisticaly I have time to play like 20 games / JAM so not being able to rate the ones I play made me not that interested to play them at all. I know that's kinda weird but that is how it is.

I have 2 more points that I will say.
1. is that I understand that the old system was kind of unfair to single dev games because they could not engage that much as a 5+ person team like ours so it is true that it was not completely balanced, and I do see that.

2. is that I understand that we can still do the comment for comment thing in any games that we want to, so the community engagement does not disappear completely, but for some reason it does not give the rush it used to. At least for me it doesn't.

Uh and an extra technical detail that I don't understand: Why can't we see the capsules of the games that we are supposed to rate? It makes choosing what to play and what not to play even from a small 5 title list very cumbersome.

Whad do you guys think? Am I being unfair here? Do I not see something that makes this a superior rating system? Any thoughts?

Submitted(+4)

I 100% agree with you! I've taken part in a few Jams already and I feel like it was always an unwritten rule that if someone rates your game, you rate theirs. I think this is the perfect system, it builds a community and rewards people who rate many games themselves. Of course it leads to a higher rating disparity, but I think that's okay - presentation in my opinion is another big part of making a game, and the "old" system rewarded people who put a lot of thought into name, thumbnail, title etc.

Additionally, the big disparity is still there! Some games still have around 30 ratings while others have just 5. Just this time around that's not decided by the effort of the team, be it by socializing or making a good game page, but purely by luck.

I see the argument for this system, and I gotta say: I even enjoyed playing games from my queue! It is true that in other jams I was "pickier", and now I got to enjoy games I would otherwise never have tried. Taking away the community aspect and making everything based on pure randomness has removed quite a bit of the charme from this Jam and left me pretty frustrated...

I hope the rating system gets changed to the old system at a later date to (possibly) get the best of both world. I doubt it though. There has apparently been a lot of discourse on the discord regarding this rating system already, with minimal to no response from anyone associated with the jam.

As a last comment: Thanks for creating this post!

Submitted(+2)

Thanks for replying, good to know I am not the only one.

Also another thing I found is this official post where this system was first introduced:
https://itch.io/t/2200921/rating-queue-for-jams-now-available-for-all-hosts?fbcl...
It seems like there is an option to unlock the random queue after rating a given number of games! So it should be used and the number should not be higher than 10 I think.

Realisticaly if there is no response on the discord in this topic I doubt it will get a reply here, but let's hope for the best!

Submitted(+1)

I agree 100%. I’m like just a student and I’ve always wished to be able to participate, play games, comment tham, and get replies and ratings. This new system is basically making my logging in everyday to see if I have new ratings useless.

Submitted(+1)

Yea the new system seems have really dropped the quantity of ratings overall in this jam. 

Submitted (2 edits) (+1)

There’s a discussion on the discord about this exact topic, including some work from user Gavrik who modeled and simulated the random voting queues over many iterations, their results suggest that somewhere between 40-100 average ratings per person is a sweet spot where you can get error rate to around ~10% before diminishing returns kick in.

I’ve rated over 200 games from the queue, including a ton of low effort stinkers and game genres I’m not super into. I have a full time job and 3 kids, so I am a little unsympathetic to concerns that 20 games is a big ask :P Especially considering ~80% of the games are 5-10 minute experiences.

But regardless, I totally agree that a threshold amount of random queue ratings, followed by unlimited chosen ratings is superior. There apparently is even a setting for this exact method. I don’t know how likely the mods are to change it this far into the voting period (or if they even can)

Submitted(+2)

For me at least playing and rating a game is just a fraction compared to writing the actual feedback. Even for short games, I tend to give the games a good amount of play time by trying to get a good score or doing a second speedier run etc. I like receiving actual feedback so that's what I give other games too. Just pure ratings and zero effort "good game" comments don't really do much for me. Of course it takes more time this way though. I've played/rated/feedbacked 13 games so far so I'm maybe projected to reach a total of 30-40.

Dang, I was kinda hoping that it would have been that FFA unlock at some queue rate threshold. But good to know that there is at least an option for it so can take that to account when joining jams.

Submitted (1 edit)

I see what you are saying.

For me personally, I didn’t upload my game for critique or feedback, I uploaded it so people would play it. The final scores are all the feedback I need or want.

It’s not that I mind receiving critique - I even enjoy it to some extent - rather it’s mostly that I really just don’t care that much. I know for myself which parts of my game are shit and which are good. I just wanted to share it with a lot of people, and most importantly of all I wanted to prove to myself i could make something I was happy with.

Submitted

Yup, of course getting people to play your game is most likely the main target for everyone. Never really considered that some people might not really be for the feedback/critique. 

Submitted

I completely agree with you.
I think that up to a certain point, it's "fair" to highlight and offer rating opportunities for games that haven't recieved many views. But I could imagine that doing so would involve some kind of reward, for example, appearing on other people's lists or something. Because in reality, I've voted for many games, still have games on my waiting list so I can't vote for games I've chosen to play, and my game still has no votes. It makes everything tedious and frustrating. 
It would also be helpful to have an idea of the number of games in our queues, because I initially thought there would only be 5 and then I could vote on the chosen games, but it seems to be infinite -_-

Submitted(+2)

I don't really often frequent these jam community tabs but specifically came to see if there was any talk about this topic now. I really see this as an inferior system too. I've been actively jamming for over ten years now always going solo. Of course teams have a bit of an edge if all members rate/feedback actively but I've found that with interest provoking thumbnail and by giving quality feedback you can kinda combat that and get a good enough amount of feedback coming your way too. But with this system seems to be pure luck and ratings really plateaued at 10 ratings after few days in and no real feedback at all. And it doesn't seem to help at all no matter how many ratings or feedbacks I give myself. Doesn't really motivate to go for quality feedback.

I was also kinda hoping that the queue would be some set length after which you'd be free to rate what ever. Sad that isn't the case. Totally agree that it'd be nice to see the thumbnails for the queue games and maybe even expand the amount to double to offer a bit more choices. Of course you can just open all the games now and pick like that but I don't see why this game listing couldn't (or wouldn't) use thumbnails as every other game listing on the whole website does so.

Submitted (1 edit) (+1)

Thanks for giving a bit of perspective from the single dev side!
Also just a random info, our game also got around 15 ratings in the first 2-3 days, and nothing since then. Very demotivating.

Oh and I just checked out what's in my queue right now, all 5 titles have 25+ ratings already. So this is clearly not the case for everyone.

Submitted(+1)

I think the random queue idea is actually a very good one, and should make the rating distribution a lot fairer.

However I agree that there should be a threshold (around 10-20 IMO) after which you can rate whatever game you want. I was surprised to find that this is not the case after rating 47 games. At this point, in all honestly, what remains in my queue is what I would rate quite low and I refrained from doing that and have stopped rating altogether.

Jam HostSubmitted(+4)

Hi there - thanks for flagging this! This should be fixed now - it was supposed to have been set to 5 like it has been in previous years. I incorrectly thought that the Rating queue unlock size defaulted to 5 like Rating queue size 🤦🏻‍♂️


Submitted(+1)

Hey Lee! Out of curiosity is there roughly some minimum number of ratings a game needs to be visible to the judge(s) selecting winners for each category?

Jam HostSubmitted (1 edit)

> Hey Lee! Out of curiosity is there roughly some minimum number of ratings a game needs to be visible to the judge(s) selecting winners for each category?

Hi! Right now, with 8858 total votes, an entry needs to get the median number of votes (14) to get the full portion of it's score. I can't find great documentation quickly on it, but this thread might be helpful.

Submitted

Uh yeah, that is good news! Thanks for the fix :)