Do you not forgive developers for mistakes, evaluating the game as a finished product, or do you evaluate the game as a concept that can become something bigger than just a game created for a game jam?
For a 10 day game jam, if you find a game that looks super polished, detailed and complete, then it probably wasn't built purely during or for this particular jam. That said I try to evaluate each entry on its merits for the time a team has to produce it. Unpolished buggy games can still have really innovative concepts which are fun to experience and I have found a couple during this jam that I actually spent 30+ minutes playing. You get aesthetically very pleasing entries that aren't much fun etc. so there is a degree of relativity but on a 5 star scale I'm thinking by category is it average (3 stars), better than average (4 stars), mind blowing or exceptional (5 stars), or less than average/very pooly implemented (2 stars). If its devoid of a criteria, thats the only time I'd give it a 1.
Similar to Dave Sutcliffe, my ratings for any individual game are influenced by other games I've played in the jam.
I judge the game as it is, not what it could become with more work. Not everyone had the same amount of time to be able to devout to the jam and not everyone has the same skills or the same size of team, I don't want to take those things into account. When a game is published the player usually doesn't know any of those details and they also don't directly affect the experience of playing the game.
At the end of the jam, the submission needs to stand on its own. If I think something could become a great game with more time and polish I may say that in the comments. But I won't score it on what might have been or could be, just what's there at the time I play it.
I've been trying to keep a relatively consistent approach to rating that's somewhat similar to Miniature Giant's approach. That being said, I did add some slight weighting based on uniqueness, displayed potential, tone. Original assets weighed highly for me, though I also appreciated the use of external assets that were well-implemented, modified, and didn't feel obvious/out of place. I also dock points if a game used third-party assets that weren't being referenced in any way. The polish piece is a hard one to evaluate on it's own though. A few of my favourites were impeccable, however they were also short and to the point, so I doubt they were things worked on prior to the jam. Now if they were dozens of stages long, it might get me wondering.
Basic metrics for me (though not exhaustive):
1- Non-existent (and not in a purposeful/effective way that goes with the tone/overall experience)
2- Fairly limited or seemingly minimal effort
3- Works fairly well and no major issues
4- Well made, cohesive, enjoyable
5- Inspiring / something I'd share with a friend
Edit: Forgot to mention - I go easy on bug issues, especially smaller ones / pieces of more complex systems. That being said, if it's unplayable for me after refreshing/starting over and comments also indicate that it was unplayable, that's definitely going to factor in.
Number of developers and their level of experience is a factor too. You cant play a game of 8 experienced developers with a level designer, grafic designer, musician, sound designer, programmers, etc. And then play a game made by 1 or more people who made their first game and give score comparing the results only. I try to have all in mind.
I'll preface this by saying that everyone should rate the way that they feel is right. I'm not claiming that what follows is objectively correct, but I do think it's important to consider.
Games released to the public don't have their reviews weighted by the size of their teams but by the quality of their work. It's not necessarily helpful to give leniency to someone because they had a small team or are inexperienced. We've all entered the jam under the same set terms and none of us have the same personal conditions.
There's at least one entry into the jam this year which someone was forced to get done in less than a day. It's super impressive that they were able to make anything at all, but I can't give them a five star rating in all categories due to their unfortunate circumstances. In my opinion that's not fair to the individual in question and it's not fair to every other person who happened to have more time to work on their own games or who was able to work as one member of a larger team.
I said that you shouldn't give scores by comparing, not that you should give 5 stars to a bad game because X person made it under X circumstances. So, not 5 stars because he explained a sad story about his life and how he did what he could.
If there is a game similar to a AAA, you will rate it 5 stars and everyone else 1 star?
And if you like a game a lot and you give it 5 stars and then you find a much better one, you go back to give it 4 stars because it is worse in comparison?
If someone does solo a really fun game, on theme and with ok graphics he maybe deserves 4-5 in those categories without need to compare first with others to give rate based on that.
There are some games in the jam that I feel could be released with little to no change, they aren't triple A titles but they're extremely well done. I didn't go back and change anyone else's score as a result of this, but I also didn't lower the score of those games if they had a larger team or appeared to have more experience than most submitters. :) But even in those cases they didn't necessarily get the highest score in all categories, I did my best to judge those individually regardless of my overall impressions of the game.
Again, I didn't said that either....
Nowhere have I said that you have to give 5 stars on everything to a game made by one person in one day due to sad circumstances.
And nowhere have I said that you have to give lower stars on everything to a game made by 8 or more experienced people.
I just said that I don't think you should take those two games and give them a rating by comparing the two. Thats all.
I totally rate by the experience that I play the game, it doesn't matter to be a smooth control, it's more about the endurance of the bad move. If you have bad move, but you will pass anyways with the intergrity of sfx, music, aesthetic and mechanics, then that game will be high rated. Some game looks fantastics, but when you playing it, the control just feel so weird or the process to win is too difficult base on the weird control, that for me will be low-rated, but it totally depends on the game.
I have different idea to mention, I think some game is well polished because some developers spends time in polishing except to extend more design, I give up the platformer to make the game more focus on action to ensure my game at least has the smooth fight experience, so my game has only fight left with some unbalance data btw player and enemies, but generally it reach the point that I feels not a bad game, then I will go to implement some new stuff, that's the trade-off. However, for some game is just too good to make in ten days, then it's definitely pre-made.
To answer the question and address some points others have made - from my perspective the comments are great to tell someone they've done something impressive for the time. The star ratings I try to keep more objective to facilitate comparisons. I don't find that very easy though as whilst we all have the same constraint of time, it's often comparing apples with oranges. Games developed with different amounts of pre-made elements and teams all dramatically alter the ability for someone to scope and adapt to circumstances, and in the case of teams can dramatically alter the man hours available. To ignore these things entirely would be for game jams to tend away from having individual developers - so I take the approach that where I am unsure if I should give a game, say, a 4 or 5 in a category then I will tend to give individual developers the benefit of the doubt.