Why I hate ratings systems in reviews, and how to fix them
John Gruber responding to Siegler’s piece on the death of the spec:
As our technology becomes more humanely designed, subjective factors outweigh objective ones. Subjective factors can’t be assigned neat little numbers ranging from 1–10.
I hate ratings systems. I use them in magazines I write for because I have no choice. But the rating is something that never satisfies me, and of any review I do, it’s the one thing that I ever regret later, even if the rating was, in hindsight, only very slightly wrong. That this happens very rarely and I still hate ratings systems says a lot.
The thing is, readers love ratings. Quite often, they’ll ‘skip to the end’ and check a rating before they read a review. A rating becomes a quick filter, to weed out the mediocre. Therefore, some kind of recommendation system is required, and one that preferably deals better with subjectivity than a numerical score.
Back in the mid-1990s, I used to read Melody Maker, and, unless my memory’s gone squiffy, I recall that it ditched numerical scores for record reviews at the time. Instead, it offered two badges: ‘recommended’ and ‘bloody essential’. That’s all I needed to know: if something was being recommended, or if I should go and buy something right now. Numerical scores sort of do the same, but they vary by publication, and they also lack flexibility. They can also be quite arbitrarily applied, depending on the mood of the reviewer, or the range can be crushed by publications with weak editors. It’s not uncommon, for example, to see gaming publications rarely use any ratings outside of the 6/10-to-9/10 range, for fear of pissing off publishers. We need something better.
Still, while Gruber argues that subjective factors “can’t be assigned neat little numbers ranging from 1–10”, publications should attempt to do so within their own ratings systems if they’re rating other factors. And that’s because quite a lot of scoring in reviews comes down to subjectivity anyway—how much the reviewer likes something; their inherent biases; any emotional attachment they have to the reviewed object. If you have a ton of categories where you rate things out of ten, why not include the experience in that? Better: bin your numbers and simplify: just tell me whether I should consider or immediately buy something, because that’s all I want to know.
“It’s not uncommon, for example, to see gaming publications rarely use any ratings outside of the 6/10-to-9/10 range,”
Much as some sites won’t let you give 0 stars to something, it has to have one!
Fully agree about the subjectivity; I’ve noticed that recently, as I’m looking for a compact digital camera – and want one with a viewfinder (having one at present that doesn’t). To me, that would rack a rating up considerably, but for others it’s a non-event (or, even, a negative, as they tend to be the larger end of the compact range)
Am currently swithering between the G12 (as my old one was a Powershot), a Nikon P7100 (my DSLR is a Nikon); and the FujiX10 (which seems very good!)
This reminds me a bit of the whole ‘Like’ / +1 debate. How do you really quantify your experience into a single score/reaction? And you need to know how many users have entered a score to really ascertain how reliable that actual score is.
I really like how Amazon reviews show the most ‘helpful’ good and bad review as well as giving a breakdown by stars. It really helps get a much better idea of how the product has been received.
The same goes for IMDB: You can drill right into the results. Knowing that a film has got 5,000 10/10 reviews from users under 18 suggests it might be the greatest Bieber film ever, but really, it’s probably not for me.
@Emma: The zero-rating has always been pretty controversial. I recall Commodore User only once using that, in something like a decade of reviews. I managed to get one thought myself, in MacFormat. I can’t recall what it was for, but it was the worst piece of software I’d ever used. I added a production note saying: “Give it a one, if you must, but it doesn’t deserve it”. To their credit, the editors left the zero intact.
@James: The Like thing really bothers me. It’s such a loaded term, and it’s often entirely inappropriate. Too many times I’ve seen something that’s hugely negative stamped with LIKE! because friends of the commenter wanted to draw attention to it. Horrible.
As for breakdowns, they are indeed useful; the problem is that so few people will ever see them.
At the other end of the scale, I remember readers discussing Commodore Format’s 100% rating for Mayhem in Monsterland. If you don’t like pixel-perfect platforming, then it’s never going to appeal to you.
So I’d say there should be a four point scale – “Avoid”, “If you like this sort of thing”, “Try before you buy”, “Go get it now!”.
@Merman: That’s what Slide to Play does. I quite liked that scale at first, but they don’t apply it particularly well, and, again, I don’t really give a crap whether something is OK-ish or rubbish: just tell me what’s worth buying and what I *really* should buy. Anything else is just an also-ran or worse, and you can tell from the text—rather than a rating—if it’s an ‘if you like this sort of thing’.
As for Mayhem, I had no problem with a 100% rating being used, given that you should use the entire scale if you have one. But I did have a problem with the reasoning, which was pretty much “you’ll never see another C64 game this good again”. Yet the game had plenty of problems common to Apex output, meaning that it really wasn’t upper 90s material, let alone 100%. Still, remove the numerical system and you have a ‘bloody essential’ badge, which covers the must-have nature of the item, without having to elaborate on that decision beyond what’s already in the text.
Iv’e never liked number or star rating systems, whether it be 10, 5, 1 bazzillion.
I hoped someone would come up with a +1 system, and they did, but they did not complement it with a -1. And that leads to another problem … what if I find your review/article/opinion just meh?
Where is my +/- option.
Where is my, its ok rating? Its ok, its neither great, nor really bad, its just, you know the internet, just meh?
I want a button next to + / – that just says meh.