It is obvious when you notice it. These comments aren't too big of a deal by them self, but if you're a first time participant and have to sit on a "nice music", when there isn't any sound, for a few days until someone who actually played your game comes allong and cares enough to write a detailed comment, I would imagine that would be frustrating and leave a bad aftertaste. It's not Ludumdare, where you get 5 ratings in the first hours, it might take three days before someone even writes a comment.
I said this in another thread before, the problem with pretty much any voting system is that it relies on fair ratings and human decency, I don't think I've seen so much concentrated dishonesty in one place before and it puts a damper on the game jam.
As you said, I've noticed plenty of 1 minute ratings, copy paste spamming every rate thread, comments that make no sense to the game I've just played and people lying about reviewing a game just to try and trick as many people as possible into playing their game.
This is my first game jam and I've quickly learned that I'd rather have 20 people rate and get some good feedback about how I can improve as a developer as oppose to 40 rates that just say 'rated'
I'd like to imagine that this is only the case here because of the jam's target audience. It's Brackeys jam and it attracts mostly young first time jammers that have completed their first game ever (since that is sort of the target audience of the YouTube channel). They are exited for people to play their game, but don't yet understand what the rating period is supposed to achieve.
They only see the strategy of rate4rate as a viable option to get peoples attention, but the only way to even notify the other jammer of your rating is to write them a comment. Usually something vague and nice (we don't want to piss 'em off after all and reduce the chance of them playing your game), to lull you into clicking on their link.
Honestly I can't be too mad at them, they are kids after all and kids are known to do stupid things. They will grow out of it by the next jam or two.
Personally I prefer the Ludumdare jam. Bigger numbers quash theses outliers and their karma system encourages you to write long and detailed reviews. I also like to look at the source code of novice and pro alike. One I like to help and from the other I like to learn.
Glad you have the right goals for the jam. In fact I'll have a look at your entry right now ;)
Hello 14, I am dad (couldn't resist the urge sorry)
Attention is the end goal. They want everyone to see what they created. They want it easy and they want it fast and what is easier and faster than copy pasting generic comments and abusing the good nature of the jammers. This one is sucky, because it robs others of valuable criticism, but is not too bad in the greater scheme.
The problem I see here is that if such obvious human spammers aren't spotted and reprimanded in a timely manner, what happens with actual bots? Those beasts have a completely different and sinister goal. Get people to download malicious software ranging from bitcoin miners and spam ads to keyloggers and data encryption ransom ware. If you get the attention of enough kids one will certainly allow the program they just downloaded to run as administrator, when it so nicely asks and an hour later all the data on the family computer is encrypted with the nice offer to decrypt it for a fee.
Well actually i can agree what you said but getting people to play your games will be difficult if you dont do the exposure thing, you can also go to other games and then comment and if they want they will see your game or not it depends on them. but if you want your game to actually get noticed from these 1,800 games or more then you have to do the rate for rate(the right way by actually playing the game) otherwise only the people from the most popular one are going to get rated and perhaps the most ratings.
You can approach it a different way though. I have played 50 games so far and between my thread here and organic traffic, I have 63 ratings as a first jam nobody. If you look at the comments then its fair to say that a lot of people seem to be engaging with the game rather than just playing it for 30 seconds if at all and provided some good feedback which is way more valuable than 60 'rated' comments.
It's far better for everyone to engage with each others games and give meaningful feedback to improve as both parties benefit. If people took the time to act like a community that wants to improve, help others improve and are genuinely interested in seeing how others approached the theme and what they were able to create then there is no way you will not get at least 20 ratings.
As much as I’d love as many people playing games we make as possible (and myself to be playing as many as possible) I think it would be interesting if every participant was given a random allocation of X number of games to review in these 2 weeks. That way everyone gets some feedback and that feedback may be more meaningful as each participant has a specific total to go through.
Meaningful feedback is way more important than ratings. I’d like to hope we all want to improve and make better games in future jams and meaningful feedback does a lot to aid that.
Yeah, I have to say that sounds like a good way to do it tbh. There is no way anyone can play through all the submissions, so they tend to just review people who have reviewed theirs etc. which could lead to biased reviews or lack of feedback. Also some of the game streams play the games live on their streams, so those ones may tend to get a lot more views than others too; a random allocation would be a good way to ensure everyone gets to play a variety of quality games and everyone gets more meaningful feedback.
It is true that a lot of high profile individuals get the lions share of attention and don't rate back (at least not with their account and they probably don't have an incognito jam participant account), just look at Dani for example. They are also usually the ones to "win", since they make games for a living. But on the other hand you have the whole least rated page where there is currently a wall of 2 ratings per game.
I do see a problem with your suggestion to give the people a random allocation of games to play. This system is sort of already implemented and and kicks of the rating period. It's the random filter. Unfortunately it then devolves into the r4r system when reviews for your game start flying in. We are subconsciously inclined to return the favor. You can't change that.
It would create a lot of problems if we were to "lock" the games and say: "here dear jammer are 20 games you need to play and rate them in 14 days".
Lets suppose RNGesus gave you the following:
More then half of what you got (and such a spread could occur given enough people) you can't even play. Guess it's overtime for the mods that will have to assign you new random games that are hopefully playable.
But what if you don't have time? Ok 20 games is doable on a weekend, but you can't expect that everyone is going to do 20 ratings, should this disqualify them and who will replace the missing ratings?
Lastly, what if you want to play your friends game. Will you have to wait until the jam is over, or will you just be forbidden to rate them?
Or what if you played a game and it was just ok. The most generic game. There is nothing bad to say, but also not a lot to praise. Will you be forced to write a comment?
Honestly the current system works ok, but dose require honesty and the willingness of people to play random games. It is important that we have enough quality raters, that play as many games as possible, but you also need moderators that keep an eye out for troublemakers. I think TheDutchMagikarp removed a bunch of unplayable games the other day for example, so it's definitely not the wild west out here.
You make a lot of good points and there are definitely holes in the "here take 20 games and review them OR ELSE" approach. I guess the point I was trying to get at was that it would be great to see everyone get some constructive feedback back from their games. Now I realise how difficult this is given over 8000 people entered and there are thousands of projects out there, and you're right, if you get an allocation that is barely playable, then it renders this a bit pointless. How about having a rating system that doesn't have a "winning goal" but instead just provides an indication as to the quality of your entry? Just ideas, and I think, as you said, this system works.
In the end we just have to hope that everyone is being upfront and honest (I think for the most part they are). In the end we're all winners, imo.
The gamedev community is always the best! Everyone here knows how hard it is to make a game and everyone can appreciate your effort and is willing to help wherever they can.
(Didn't expect a discussion board about cheating to end up in such a wholesome thread, but I for sure don't mind some positivity)
I think ratings should only be possible through the 'play random games' button. Each game will still have a jam page where people can comment to leave feedback, but the only way to get ratings is by your game randomly being played by people. I think this way an even number of people will rate each game, and you can still share your game with each other for feedback.
Oh yh that is correct. I guess if they did implement a system like that, instead of pure randomness, there would have to be some way they favour games that haven't been seen by many people yet.
But my main point is that rate for rate threads and ads on the discord server should not affect the number of ratings people get at all.
I really liked the Ludumdare karma system. Taking the number of your ratings vs the number of ratings you got, with games below 20 rating getting preferential treatment. They also gave heavy emphasis on comments to boost your placement. All this was shoved in some formula to determine the order of games and I must say it worked quite well.
They did write that they will keep an eye out for cheaters, but it's not like it was too clear on what they do or how they do it (which is understandable). Now that I look at my LD page, they specifically write karma for feedback, so that would imply you only give a boost from your comment if the creator liked it. That would actually be pretty darn great! You would have to tell the peps to only like helpful comments and not like the the "nice game rate mine plz" and it would instantly curb the issue.
My game is an EXE and I have wayy more downloads than ratings. There is no correlation, when in this there should be. I first thought, "maybe it sucked so bad not to rate) but was unaware of how this all works, being my first jam. Then a game makes it to the next round, do the Jam creators begin to judge?
On my first jam (LD46) the ratings actually matched the downloads quite well.
I think this has to do with the 8000 participants that didn't finish a game. Maybe they are the shadow players.
Anyway regarding the rating, there is no next round. GMTK's jam has one where he personally plays the top 100 to pick the 10 best, but he also gives out goodies and want's to make a good video, so it's understandable.
Jams usually are participant rated and there are no judges (the hosts are the secret police that are supposed to boink the cheaters and foul players). Full democracy here.
You're so right! On my first gamejam i was so happy about the positive feedback - "great", "awesome", "nice game"... But now i can't read it anymore. I love to get some criticism, learn from others and get some new ideas. But the rating system will not change. The best way to get some real feedback would be to create a small and active rating/gamecreating community where everyone does their part with heart.
I wouldn't be surprised if they have such as system on itch.io since they host so many jams. In a database (which this whole jam is in essence) a simple cluster analysis (common method to determine insurance fraud) would uncover every suspicious transaction (in our case rating) instantly. I simply don't know if such a tool is available to the hosts, since I never hosted a jam myself and have no clue about the back end.
But I would never implement a hard timer. Not only would this give you tons of false positives, it is just going to make some people pissed.
Well what can I say, accept that I am that guy that reads the license agreement in full before installing the software. I try to improve, I really do.
Since I have nothing better to do and I want to improve my c# I will actually try to make a monitoring API. A rudimentary one, but we don't need a lot.
Well, I feel like I need to explain myself (for my own sake). I'm currently rating games that people submit to me in a thread. I saw this as a necessary evil beacause before that I hade 3 reviews. T H R E E. I saw this as the only viable way to get rates when not being famous or particullarly lucky. Btw I give people feedback, these are not empty ratings. Also I saw a guy that made about 5 r4r threads at the same time. The same guy also submited his game to 8 different jams. Also the game is Android only so I doubt many people played it, yet he still recieves ratings. Also the "most rated" sort option allows to snowball for the more popular games and leaves other creators with no way to get anyone to play the game. Something needs to get changed. Idk what but something is wrong, I can feel it. Cheers :)
Don't get me wrong. There is nothing wrong with rate4rate. We are only talking about people that abuse this mentality and how one should put them in their place. I strongly believe that a single massage from a mod calling them out would make them stop instantly and if not there is always the ban hammer looming overhead.
We like the illusion of justice to be kept.
On the change you're asking for: mods. What are they doing? To whomst one shall address a letter of electronic format concerning such regards? I don't want to press "report submission", since the game by itself is fine and this feature is there to protect others from viruses and inappropriate content. I don't want to press "report massage", since that would require going thru long timelines and would result in you getting investigated for trying to cancel a fellow creator. I would like a button like "spammer" or "report unsportsmanlike behavior" with a dedicated mod investigating. Currently I only see thedutchmagicarp doing stuff here (removing unplayable entries a day ago) and the others are probably on discord idk.
The only way to report spam is to click report post>spam. A sticky post from someone high up would probably do the job. Especially when you let people know that they can be flung out of the windows when behaving a certain way. Also are the same peoblems present on GMTK Jam? And if not, why do you think that is?
Hey I'm gonna give some perspective from the organisers side here.
So first of all, this jam has an insane amount of participants compared to our other jams. Our other jams were also big, but this jam has about 3(!) times the participant count of the previous jam. The submission count didn't grow as much, but still over doubled compared to the last time.
This means there are a lot of new problems surfacing. For example, the dedicated jam channel on the server has been pure link spam for the first day or two. In the past, we didn't have to do any kind of separation for promotion of games and discussion, while it was definitely required this time. Is that just because of the increased amount of participants? Or maybe the mindset of the participants? No clue, we don't have a way to find out either.
On the Discord server, we are easily able to solve this. We are familiar with the moderation tools and able to customize quite a bit. We have our own bot and if it's really needed, we can make the bot help us with keeping the channels clean and orderly.
The itch.io community tab is an entirely different beast. I think I can speak for all organisers here when I say that we have almost no experience using them and that's not even talking about the moderation features. Learning to use a new set of moderation tools while everything is already a giant mess (yes I admit, the community tab is currently a disaster) is a bit of pressure and takes time. It's currently summer break for most of the staff members so some have trips planned, volunteering work, or other activities during the day. For me, after working on the jam for a week, energy levels are definitely low and not really being recharged with the heatwave going on. This also has it's effect on how we are moderating the community tab, for example.
Personally, I don't have a problem with rate 4 rate being a thing. I think it's good to encourage people to play eachothers games and give feedback. I do agree with the concerns you bring up regarding people just rating 1 star without even downloading/launching the game, just to comment "good game rate mine too thanks".
(This next bit is what I believe to be true, but if Leafo is reading this he can maybe give a little better explanation as he knows how it works behind the scenes..)
The itch rating system has a weighted rating, which means that the ratings of people that just spam 1 star on everyone will be counted less than the rating of a person that gives honest ratings to everyone. This should at least take care of part of this problem.
There is still the problem of people just commenting on games saying "nice game". There is no way for us to block this kind of behaviour. There are too many ratings and comments coming in on a daily basis to moderate all of them. There is also no way for us to know when the comment is a genuine one. Maybe the person commenting really thought it was the best game they ever played and they don't have any criticism, maybe it's just a rate farming spammer. Of course we can check the history of the user, but that's another action we would have to take per rating, consuming more of our already limited time...
Then there is the rating queue idea. GMTK jam had a rating queue where you first had to play x amount of games from your queue, before you were able to rate anyone. This is something we also wanted to do for our jam, but I think something went wrong in the backend of itch (it's not a released feature yet, we had contact with itch to see if they could set it up for us and it should've been set up correctly). We don't want to keep people locked to their queue completely, because I know that I want to rate the games from my friends freely. If I would have to find them by pure luck... That would be a disaster! It would of course help with people like Dani getting insane amounts of ratings, just because they are a content creator. I have considered talking to him to see if he would be ok with not being able to win because it's not fair with his community behind him, but at the same time it's not fair to do that either! If he does win in all categories (or any category), we'll see how we can handle it. Especially because there are no prizes on the line, it's a lot easier to just also mention the 2nd place games too.
I am definitely going to bring up some points about the community tab (and post-jam community in general) in the post-jam organisers reflection chat, so we can hopefully make the experience a better one next time. I will also discuss if we should block the rate 4 rate threads completely and start cracking down on them more during this jam, or if we see another solution.
I think I have talked about the most important issues brought up in here, let me know if I missed any.
TL;DR: We don't know how itch community works, time is valuable and limited, we are going to reflect on the current state of the rating period and hopefully improve for next time.
Thank you for your insight into the inner workings. It cleared up a lot of questions.
I would like to offer some ideas regarding some issues you brought up
I hope that you will find a way to mitigate the current mess and prepare better procedures for the next one.
who are handling community post reports? I reported some posts in appreciation thread some time ago because people are just spamming their own games which is not in the spirit of the thread and is quite annoying to be frank, the entire community section is already filled with these posts and threads and we can't seem to get one thread of peace to appreciate the work others have done.
I would absolutely be willing to dedicate a couple of hours to moderate a stickied thread dedicated to recognising other peoples work next jam.
That is completely fine. I had a look at your timeline and while it is sort of generic you aren't copy pasting things and leaving dozens of comments.
There is no secret sauce for judging. The option of a novice is as valuable as that of a pro. In the industry your games will generally be played by people who have no idea what it takes to make a game. All they can tell you is if it was fun or not. In the jam it is your job as a rater to focus on things you have noticed and if you just can't find anything that could help the jammer improve it might be best to just move on to the next game.
So I have made the first step for my cheater detector. A simple text comparator!
Here is the sourcecode:
https://gist.github.com/Voidsay/d84a64fc94e79a31c1a005da1245c2cc
It will be able to detect copy past comments and slightly misspelled as well as slightly varied comments.
Next I will figure out how to go thru all participant comments and compare them to each other to calculate a final similarity score for each participant.
Hopefully I will then be able to see a clear difference between normal commenters and dirty dirty cheaters.
I just posted a new topic where I ask if people can help me reach 20 ratings (I needed 3 more).
Someone answered it and wrote in the comments that it is a "cool time killer". My game is a puzzle stealth game. I have no idea how it could be a "time killer" and chances are that he didn't even play the game, and didn't even read what the game is about. I was really afraid to make that post and I actually asked there for more detailed comments in order to avoid that it would become like that but people just don't read the thread.
It is just so disheartening.
Looked at the dude in question. Exactly what I need to test the algorithm. This is the type of comments I want to boink with my rubber hammer of justice.
I expect that it will take a couple more days until my API is fully operational, since I have never done one before. Plus I might need to get extra permission from itch.io and approval from the jam host.
I am not writing a full auto ban hammer. Just a tool for the mods to use to look in the right places. As far as I understand there is only thedutchmagikarp doing all moderation and support on this jam. And he dosen't even have proper moderation tools! Thats what I want to change (and also increase my qualifications, programming is pretty cool after all)
So I have been working on what I am from now on calling the "comment uniqueness index rater" (will be referred as CUIR in this post) all day.
Turns out interacting with a website is actually pretty easy! Almost as if people have been doing this for decades and created special tools, libraries and packages... *big hmmm*
I am now able to scrape the jam page for all the usernames, the game names and the urls. Following the links to the game page and collecting all comments and running them thru my CUI calculator should be a breeze now and will be done by tomorrow.
The whole functionality will be as follows:
Features:
Flaws:
When everything is done and in a presentable (and hopefully working) condition I will release the sourcecode on my github for you to scrutineer. If we're lucky the program will do what it's supposed to do by the weekend.
I have been working steadily on the project. Currently I am battling the mysql database I myself created. The whole deal with the utf8 character set really messes with everything and displaying these characters in ascii just looks awful and is unreadable.
I am setting up the database and will probably export it into an excel file as well, so that everyone can check it out. It just takes a little time.
So I have finished my program. Unfortunately the algorithm didn't give me the spike in the cheater ratings that I had hoped for. Some other more advanced process needs to be found to catch 'em all. More hit or miss. The only thing that might be interesting is to lookup the people with more than 100 posts under games. I don't trust 'em, but the algorithm says they're cool.
In any case I have learned a lot and will refine my skills further.
If you are interested in your "performance" you can lookup your own score here:
https://itch.io/jam/brackeys-4/topic/931373/i-rated-you-comments-see-the-results...
I tried to write a little something something to detect such behavior. Sad to say my cheater detection attempt didn't work out quite as well as I hoped.
The ratings need the comments of professionals. Those are the most valuable, since they know what they are talking about. Sadly they are also the busiest and can't rate all day every day.
Suggestion:Use the itch.io server-side API to check out the state of every user on every game by using a GET Request,then process that information and store it in the mysql database and determine if it the state is not_viewed maybe set to true in the database then you can make an event maybe when somebody is commenting when not_view is set to true maybe do a POST Request to post a send a message the program caught maybe saying something along the lines of:'You cannot rate before viewing the game'
What are you doing in this dead thread?
There is no problem with the scraping itself. My c# script works perfectly fine and the database fits all the data well. The problem is the analysis itself. I hoped that the Jaccard index would be enough and create a bump in the bell curve somewhere in the lower score indicating an anomaly aka the cheaters. Unfortunately that didn't happen at all. In fact some confirmed "cheaters" scored close to the middle.
I kind of expected the algorithm to fail, since it is the barest of bones. All I got from it was a little scraping and database experience as well as a sort of inaccurate comment uniqueness rating. There is another thread about this tough.
Here's another solution:Use the itch.io server-side API to check out the state of every user on every game by using a GET Request,then process that information and store it in the mysql database and determine if it the state is not_viewed maybe set to true in the database then you can make an event maybe when somebody is commenting when not_view is set to true maybe do a POST Request to post a send a message the program caught maybe saying something along the lines of:'You cannot rate before viewing the game'
I had a look at the API, but there are a couple of problems.
As far as I am concerned the API is mostly intended to do fancy things with your own games. Quickly changing prices making an auto reply bot things like that. It is complete overkill for simple get requests.
It also requires the users to accept one thingy (I completely forgot what it's called). Long story short, I can't force this and it doesn't give me anything that I could use for my purpose.
I am unsure how I would even go about implementing your post suggestion. Write a silly comment? I don't have access to the server, I can't make a popup and lock the ratings.
Also I am new to APIs and the documentation is way too hard to read/implement. Maybe I'm just dumb again and its super useful, but my workaround is fine.
I don't know about requesting the state of every user and game constantly. Firstly it took me about half an hour to collect all data by itself, too long it would let peps slip thru (I might optimize it to make it faster, but it mostly depends on the internet connection that I can't change). Secondly I think that itch.io would kick me for attempting to dos their server
Also the automatic ranking system already nerfs suspicious ratings. They look at all the star ratings you gave and shrink their wight if they don't follow the bell curve. This means that "randomly" clicking (humans can't click randomly) on stars or review bombing in disregard of the game you will probably result in an uneven distribution, which in turn results in your ratings having zero weight in the final score. Cheating that might be more difficult than actually playing the games.
Besides this isn't really what my cheater detection was about. It's more about low effort comments copy pasted to dozens of games. That's what I wanted to detect. People that just want to get you to play their game and don't want to put in the necessary legwork.
I have a YouTube channel,if you're planning to program in other programming languages at:https://youtube.com/channel/UCqj9jELS5ayGGl8WES0x70w
