In an editorial today, Engadget’s editor-in-chief, Tim Stevens, declared that the era of numbered scores on product reviews was going away in favor of bullet-pointed pros and cons and a summation in under 140 characters. It’s hard to remember a time before their various reviewed gadgets were adorned with those color coded numbers… nah, nevermind, they were implemented a years ago as one of former EIC Topolsky’s last big moves on the site. Still, the use of numbers to denote, in an abstract way, the quality of products is of much debate among many websites. At the end of the day, I think they’re a necessary evil and I think Stevens is on the wrong track here.
On the Justification of Numbers
While I sat in line to totally get autographs from Adam Sessler and Morgan Webb when we went to E3 a few years back, I got to chat briefly with one of the producers of G4’s video game review show X-Play. I complained mostly that their five-star system sucked, because they basically give every other game a three out of five, leading to the impression that everything is average. I told him that I loved the PC Gamer percentile system and that over the course of several years, I had built up this spectrum of how the magazine felt about games and I was able to make better judgments about games because I had learned their system. Gary Whitta, editor for a time there, had encouraged his editors to use the ‘whole spectrum’, so bad games were probably 50% and really bad games were down toward the 20s and 30s. The producer, in turn, asked me what the difference really was between a 72% game and a 75% game, to which I had no answer, because it wasn’t the individual scores that meant much, but rather the sum of the scores that did.
Anyway, the point I’m trying to make is that I enjoy numbered scores. We at FleshEatingZipper use a numbered system for everything that isn’t hardware, which exists in its own continuum. When Kelly and I worked on our own movie review website a few years ago, we settled on a three tier cold/warm/hot because it was based on genuine recommendations to people. If someone has ever asked you if they should see a movie, you’re gonna say one of three things: Yes!, Maybe (with specific conditions), or Nah. That’s something we wanted to bring over to our hardware reviews, but with only three in the pipe, it’s hard to see how well it’s reflected.
The thing is, even when X-Play does a three out of five, Metacritic considers that a 60% and it might not make much sense individually, but when composited by an aggregator, it makes a lot more sense in the wide world of reviews. So, then Engadget….
Right, So What Engadget is Doing
On its face, Engadget seems like that rebel standing in front of an indifferent crowd with a leather jacket, twirling a chain and some of Stevens’ points seem legit. Let’s just work down the editorial that explains why he’s decided to do it, his first complaint being with the aging of content:
More than that, the system attempts to scale an already established number based on the relevance of the device it was assigned to today. That’s of course because a review score never changes — it never expires, has no shelf life, but is still distinctly perishable. Products that were deserving of a 8 or 9 last year probably wouldn’t receive the same score today. Those numbers are stale.
So a number makes a review stale, but the actual review, which he seems to be an advocate of here, doesn’t? How does giving a cutting-edge Android phone a high score now make it any more impervious to the rising tide than the fact that you spent a paragraph praising its amazing display when some new tech is right around the corner. Oh, that 3MP shooter was so hot in 2008, doesn’t make it good today just because you spent more time on it. In this case, a score contextualizes in the time it was released over a bunch of text gushing over its now-obsolete innards.
A review score is a number, a single digit (we never did cover something worthy of a pristine 10) that gives a final, conclusive rating of the overall quality of a given device. […] A review score takes all that information and more, the subjective and the innate and the substantially complex plus the objective and the concrete and the easily comparable, and tries to rank it against the entire galaxy of other devices that may or may not compete directly or indirectly with it.
Again, I think tossing out the number or any at-a-glance mark marginalizes the review. Tim admits that we’re all super busy, so he concedes on the “Bottomline”, which is the aforementioned pros and cons, but now you have to drill into each article to extract that knowledge. If Engadget gives a device a 3, I know I don’t even need to read further. A 7 or 8 and I’m going to skim it. Higher and I’ll give it a full read. Their sidebar widget is now useless because I can’t get a feel for how they feel about gadgets at a glance. There’s nothing disenfranchising about boiling your feelings down into a single-digit number because we can all have gut feelings these things.
His niggle about the fact that they never gave anything a perfect score is silly. We don’t give out those scores either, those are for special things. But he makes an interesting point about how subjective those numbers are anyway, which isn’t really an argument against numbers, but making sure there’s plenty of review to back it:
A review score attempts to take a couple-thousand words worth of exposition that deeply analyzes the many and myriad features of a given device or service and boil all that down to a single digit. It does an incredibly poor job of it.
I don’t think a number is based on thousands of words, but again, a gut check. It’s a different kind of review and that’s what a lot of people, myself included, want from a publication. Maybe I don’t need your specific pros and cons, maybe I just want that ‘Get it!’/’Maybe Get It!’/’Don’t Get It!’ reaction. A seven out of ten iPhone review means a lot more to me than a ten out of ten write-up, but I don’t need to run through your site reading tons of prose to do it. You may write about a product negatively, but your gut score, that seven out of ten, might be how you actually feel. Sometimes it’s in how you do it.
I think it’s myopic that Stevens decides to cast away one huge chunk of his review structure on principle. I’ve read enough reviews to know that you’ve gotta have a vertical slice of everything to really get people interested in your opinion. Reward people with three thousand word disseminations of the latest in tech, don’t punish people with them.