Your assumption is actually incorrect. Games with fewer votes do get an adjustment - you could call it a penalty, but what it does is balance the ratings so the games with fewer ratings aren't automatic winners.
The formula for jam ratings was explained by an admin. According to them, the formula is:adjustment = sqrt(min(median, votes_received) / median)
This means that the adjustment for a game that got 19 votes in a jam with median of 20 is = 0.97467, about a 2.5% reduction in average score, eg. a 4.5 rating would go to 4.386 for the purpose of comparison during ranking.