I teach a course in game development. As a part of the course. I asked my students to upload their developed games to itch.io. I promised bonus points to students who will have the largest number of 5 stars rankings. I suspect that some students opened fictive itch.io accounts in order to artificially give a lot of 5 starts ranks to their own game. Is there a way to check this? E.g., is there a way to check whether many different accounts were opened from the same IP adderss?
The three leaders currently are: https://avihuoshri.itch.io/test-drive and https://aripash.itch.io/five-balloons and https://shaykeshok.itch.io/gold-minner .
I know that they spent a lot of effort to have their friends register into itch.io and rate their game, which I think is legitimate.
What I would like to know is if any of them created a lot of fictitous accounts (e.g. from the same IP address), which I think is illegitimate.
Thanks
We can take administrative action against accounts if there is fraudulent activity going on, but we can not reveal private details about accounts like email addresses and ip addresses. In the future I recommend not using something for your class that enables students to cheat so easily. A good alternative could be hosting a game jam just for your students, and only allowing the people who submitted to rate the entries to prevent cheating.
I can tell you that we did just end up suspending a lot of accounts associated with those pages.
Thanks. It's been a hard semester, we could not have exams on campus due to COVID and had to find creative ways to encourage students to work. I did not think they will abuse it this way.
Question: when an account is suspended, are the ranks given by this account to specific games removed from the rank-count of the games?
The displayed rating on the project pages will now reflect any accounts that were suspended, but we can’t 100% detect all cheating, so I still believe that this metric for rewarding someone is not the best idea. We have a similar issue with people who host jams and have the voting set to public. It’s just too easy for people with more resources to completely push out anyone else.
Thanks a lot. Did you also check this page? https://aripash.itch.io/five-balloons
I totally agree with you that this was not a good idea for grading. I was too naive.
Another question: Is there a way to see how the number of ranks of a game changes over time?
It looks like more fraudulent accounts are being created. At this point I honestly recommend you tell your students that you can not provide any incentive or bonus for ratings, since they’re going to keep cheating.
If we keep seeing fraudulent activity that is negatively impacting the platform then we may have to take more strict administrative action on the accounts. I’m seeing hundreds of fake accounts being created a this point.
Thanks
I just want to add: Yest we can detect the people who are cheating in a very obvious way and remove those votes. But it’s hard to detect with confidence for smaller scale cheating. You will have people who might try to create one or two additional accounts to give themselves a little boost, which is unfair for those who don’t believe in cheating.
Additionally, rating systems only work at scale: when you have thousands of independent data-points. That will help eliminate any noise and give you a high confidence that the average rating represents the project. In your case, there’s probably a small chance that the projects for your class are going to be receiving a large number of independent ratings. The number of independent people seeking out school projects on our platform is not high. So you’ve effectively told your students to go reach out to their network of friends & family to get them to vote for them. You’re rewarding the students resources, not their work.
This is a completely unfair way to handle a school bonus, so I strongly recommend you find another way to handle it.
I'm not claiming to be an expert in education or anything here but... is rating your students based on the number of 5 star reviews really a good criteria to determine their skill or abilities?
Even if you exclude the ones who set up sock accounts (and it's very easy for anyone with even marginal technical experise to use different IP addresses), are you not just ranking your students in terms of how many compliant people are in their family and friends circle?
I agree- but, my issue is- if someon posts a game, and knows the criteria for success is to get as many five star reviews as possible, then the natural progression is that they'll try to get all their friends, all their family, all their work colleagues, anyone who owes them a favour etc. etc. to give them a five star review- regardless of whether they've even played the game.
Essentially, it becomes a popularity contest, like Americans selling cookies on behalf of their children, and the quality and effort put into the game becomes completely irrelevant.
Fair enough. I've got to say, I have a lot of questions I'd like to put to the OP but... nevermind- not my problem ;)
Slight Edit: I'm thinking if the OP thinks that checking IP addresses will be enough to stop fake reviews then... maybe not? I mean, we've had some weird reviews from some interesting people but never anyone just saying "great game" or "fantastic game" on the only game they've ever reviewed! Ok, I'll stop now, I promise ;)
The grade in the course was based mainly on the development process of the game itself during the semester. The "marketing exercise" was for bonus points. I promised 10% bonus for the first place, 5% for the second, 3% for the third. I did not believe that student would cheat (and risk losing the entire course) just for this small number of bonus points.
I also teach game development courses. While I appreciate what you are attempting to do by teaching "marketing", I would recommend against using ratings from random itch.io players as a metric for assigning bonuses.
First, itch.io is a skewed audience. Look at the kinds of games that are highly ranked in most game jams and you are going to see a strong preference for certain types of games (eg: remakes of classic action/arcade games with innovative twists). One of the world's most profitable games is a social bingo game targeting older women, but I doubt that would do well here. Similarly, "Style Savvy" was a fantastic DS game about fashion, but I doubt it would fair well with itch.io's indie-focused audience. By letting itch's audience be the judge of your students' games "marketability", you are incentivizing your students to only think inside the box of this one specific audience rather than opening them to the possibility of appealing to more diverse backgrounds.
Second, you have no idea what metric an individual stranger is really using. A player may not rate a great game highly simply because the game's genre is unappealing to that player; so even if the game were the pinnacle of that genre, it wouldn't fair well.
And, finally, think about how many revered artists today were reviled in their own time. Adherence to sound design principles and intentionality is a good way to judge art because it is timeless. Judging art based on popularity is fleeting and may squash a future revered artist who is simply misunderstood by the masses of today.
If you want to teach your students how to design to appeal to an audience, why not pick a diverse handful of specific people to "judge" their games. Give the students a list of each judge's preferences and let the students design a game targeting 1 specific judge (their choice) or a have the students create a different game for each judge. This would not only encourage learning both good design principles and audience targeting, but it would also present an opportunity to teach about issues like accessibility - especially if you pick judges with special needs (like a colorblind judge or an elderly judge with arthritis).
