Why I hate ratings systems in reviews, and how to fix them

John Gruber responding to Siegler’s piece on the death of the spec:

As our technology becomes more humanely designed, subjective factors outweigh objective ones. Subjective factors can’t be assigned neat little numbers ranging from 1–10.

I hate ratings systems. I use them in magazines I write for because I have no choice. But the rating is something that never satisfies me, and of any review I do, it’s the one thing that I ever regret later, even if the rating was, in hindsight, only very slightly wrong. That this happens very rarely and I still hate ratings systems says a lot.

The thing is, readers love ratings. Quite often, they’ll ‘skip to the end’ and check a rating before they read a review. A rating becomes a quick filter, to weed out the mediocre. Therefore, some kind of recommendation system is required, and one that preferably deals better with subjectivity than a numerical score.

Back in the mid-1990s, I used to read Melody Maker, and, unless my memory’s gone squiffy, I recall that it ditched numerical scores for record reviews at the time. Instead, it offered two badges: ‘recommended’ and ‘bloody essential’. That’s all I needed to know: if something was being recommended, or if I should go and buy something right now. Numerical scores sort of do the same, but they vary by publication, and they also lack flexibility. They can also be quite arbitrarily applied, depending on the mood of the reviewer, or the range can be crushed by publications with weak editors. It’s not uncommon, for example, to see gaming publications rarely use any ratings outside of the 6/10-to-9/10 range, for fear of pissing off publishers. We need something better.

Still, while Gruber argues that subjective factors “can’t be assigned neat little numbers ranging from 1–10”, publications should attempt to do so within their own ratings systems if they’re rating other factors. And that’s because quite a lot of scoring in reviews comes down to subjectivity anyway—how much the reviewer likes something; their inherent biases; any emotional attachment they have to the reviewed object. If you have a ton of categories where you rate things out of ten, why not include the experience in that? Better: bin your numbers and simplify: just tell me whether I should consider or immediately buy something, because that’s all I want to know.

November 15, 2011. Read more in: Technology

6 Comments

Death of the spec in tech reviews

MG Siegler for TechCrunch:

[Tablet] makers may be largely failing because they’ve sold their soul to Android and are now just in the middle of a spec war, which no one can win. I’m gonna go one step further in that line of thinking: the spec is dead.

I’ve been arguing this for years to anyone who’d listen, but with techies, specs have always been the only means of differentiation. That’s because they are solid numbers, rather than subjective traits. And when people wanted to buy tech kit, they roped in a techie, who’d spew numbers at them and come up with a ‘winner’. Most reviewers did (indeed, often still do) exactly the same thing.

I mostly put that paragraph in the past tense, because, as Siegler suggests in his article, a change is finally occurring, which has been driven primarily by Apple and Amazon. When the iPhone came out, its specs weren’t the best, but it offered an overall experience superior to the competition. This started to seed the idea that buying a gadget was more than just about numbers—something Apple had often been unsuccessful in promoting with its Mac hardware versus PCs. With the iPad and Kindle, this became far more apparent. Apple’s tablet lacked features the competition tried to ram down people’s throats, and the people didn’t care; Amazon’s Kindle was a simple, ‘underpowered’ device compared to, say, the Nook, but was perfect at the task for which it was designed for. Again, the experience was winning over spec lists.

This week, we’ve seen Apple’s PC growth figures, which outpace pretty much everyone else in the industry (the only notable exception being a big spike from Samsung in some European countries), and much of that success is down to the MacBook Air. On paper, that device looks overpriced and underpowered. Its specs are weak: slow processor, not much storage, and so on. But the experience is, according to those who own one, amazing. The machine is light and feels great; it’s fast, due to a combination of factors, down to… well, down to stuff that no-one really cares about any more. And that’s the point: perhaps we’re finally getting to a place where people will walk into a store and buy the best experience rather than the device with the best numbers.

 

November 15, 2011. Read more in: Apple, Technology

1 Comment

Amazon Prime UK versus Amazon Prime USA

Amazon Prime has been a regular feature on my Twitter feed for a while now, but its reasons for appearing are markedly different depending on the location of the commenter. If you’re in the USA, it seems you’re continually bowled over by Amazon Prime; it is, in short, so great that it’s being used as a service to subsidise the Kindle Fire tablet. In the UK… well, things aren’t so good.

Americans are often quite surprised when I complain about Prime on Twitter, while Brits are often shocked when Americans bang on about how great it is. Here’s a comparison of what you get:

Amazon.com (cost: $79—about £49—per year)

  • Two-day shipping on millions of eligible items
  • Unlimited streaming of a ton of movies/TV shows through Prime instant videos
  • A free Kindle book to borrow each month

Amazon.co.uk (cost: £49)

  • One-day shipping on millions of eligible items

Even the UK’s sole advantage (faster shipping) is purely down to geography, given that the country is significantly smaller than the US. Additionally, I found Amazon Prime’s reliability and value went out of the window. I’d been a Prime subscriber for some time, but in 2009 the service was extremely creaky around Christmas. In 2010, it was beyond a joke. Amazon had to deal with a harsh British winter, to be fair; but that doesn’t excuse the company’s shift from trackable services to just flinging items out using Royal Mail’s standard first-class service. I therefore had the absurdity of receiving a small book and DVD by next-day courier, and two Kindles rattling around in the system and, by some large measure of luck, finally arriving well over a week late on the last Royal Mail delivery day before Christmas.

Amazon’s response was also pretty telling: it pretty much told me “tough”. Goods are sent by whatever service Amazon chooses to use, and the value of the items is largely irrelevant. Into the new year, things were no better. Pretty much everything now came via Royal Mail’s first-class service, and so my £49 per year was losing value fast, and, worse, Prime goods were eventually taking longer to arrive than those shipped for free from rivals like Play.com. For my winter’s troubles, Amazon made a big fuss about extending my Prime subscription by a whole month, but it lapsed in April and I’ve no desire to start it up again. (I also note that Amazon’s dispatch times for ‘free shipping’ changed from fast to sluggish over the past couple of years, presumably to push people into upgrading shipping. Cynical.)

I know other in the UK have had similar experiences, while some have found Prime totally robust. But it makes me question whether those burned would ever return to the service even if it started offering the extras Americans get. But while Prime UK is merely ‘possibly faster shipping, if you’re lucky’, it’s not hugely attractive and is another case of an international company providing some of its users with a banquet while the rest have to do with a sandwich.

November 11, 2011. Read more in: Technology

3 Comments

Will tablet computers kill handwriting? Or: why investment in technology for schools is a good thing

The Metro is today running a piece called Will turning to tablet computers to educate pupils kill handwriting?, which offers some interesting takes on the future of education and handwriting.

If you were a teenager sitting at school and your teacher asked you to choose between a Biro and one of the most sought- after devices in consumer history, which would you go for?

My assumption is that teenagers (and other pupils, for that matter) won’t necessarily get a choice—they’ll get the tools that are best for purpose, making allowance for budget and politics. In the same way that I find it unlikely 100 metre runners will be offered, say, a motorbike, teachers won’t be offering a shiny tablet for all school tasks, or asking a pupil to choose between technology and traditional media.

Both are key tools in a child’s development but there are concerns that handwriting will be drowned in a new wave of technology in schools.

The use of ‘drowned’ is very loaded. But here’s the brutal reality: most people in the UK don’t write much these days. I mostly type words, for both work and play; now and again, I’ll make written notes, but I genuinely cannot remember the last time I wrote anything long-form traditionally.

‘I can’t understand why the schools are doing it,’ says Sarah Mooney, principal of the London College of Graphology, who believes writing promotes creativity in a way a computer cannot.

Person with vested interest in the science of writing and relation to psychology in ‘technology that makes handwriting less important is bad for people’ shock! And while she might believe writing promotes creativity in a way a computer cannot, I’d say that’s rarely the case these days. Most people who write for a living are thrilled by the sheer flexibility computers offer. In my case, I can rapidly bang thoughts into WriteRoom or Scrivener and then mould and shape them rather like a sculptor working on a piece of rock. When I used to write using only paper, the process was slower and I’d be frustrated by errors and editing. At school, this was even worse, since we were encouraged to submit final English essays with no errors at all, or we’d be marked down. This turned a creative pursuit into laborious drudge work, which the computer typically makes significantly less painful.

Also, if we look at very young children, the simple act of holding a pencil and learning writing doesn’t come entirely naturally, whereas interaction with something like an iPad is far more intuitive. Children have stories to tell, and enabling them to do so before they’ve mastered writing unleashes creativity—it doesn’t restrict it.

‘People won’t be able to write eventually. It needs practice. I think that it will affect people being able to express themselves properly.

I don’t fully understand why Mooney thinks this is an either/or case. Yes, digital is becoming more important, but kids on our street still play football, despite EA’s FIFA being available for every modern console known to man. Kids still love making music with real instruments and learning to play them, despite digital music tools existing. Kids love painting, despite the prevalence of Photoshop, Painter, Brushes and other creative apps. More to the point, core physical skills aren’t going to disappear. Schools aren’t going to ban children writing and instead force an iPad into their mitts. Instead, physical skills and digital skills will go hand-in-hand, and pupils will use the most relevant skill and medium for the project they are working on. Again, this is beneficial, not a drawback.

‘It’s money being spent on the wrong thing. I think it should be spent on the teachers and less on, shall we say, material things.’

Mooney makes the assumption that this is about glamour and having more possessions. It’s not. I last year spoke to Fraser Speirs about his project, where every child in his school now has an iPad. He believes this has saved the school money through the shifting of purchases, empowered students, and also, importantly, made things better for teachers, not least in them being able to engage students and be more creative in terms of teaching. That to me suggests an investment in the right kind of technology is a good thing, not wrong.

I’m glad to see that the Metro’s article does provide balance. It next talks to Sara Davey, headteacher at Mounts Bay Academy comprehensive in Penzance, who says giving every pupil an iPad will make learning more interactive and democratic:

‘We actually think it’s the future,’ she says. ‘Learning used to be, particularly in IT, an individual at a monitor. What the iPads allow is collaborative group work and students chatting about ideas, brainstorming.’

And Speirs is also quoted, offering the kind of future that would make Mooney’s toes curl:

[He] makes it clear handwriting has not been discarded—‘we are not a paperless school’—[and] would welcome a time when pupils do exams on iPads or computers instead of using pen and paper.

‘I think that’s something that’s got to come,’ he says. ‘It seems a logical end point. Handwriting already is a dying art and it’s you and I who are killing it, because adults are not handwriting.’

However, he notes that writing itself isn’t going to die through the use of touchscreens:

‘What you might see is a generation of kids growing up who are as good at writing with their finger on a touch screen as they are with a pencil.’

It’s also worth noting that while everyone’s touchscreen crazy right now, there’s a good chance devices may offer more physical interaction in the future that dispenses with the generic ‘pictures under glass’ UI that’s now so popular. Bret Victor talks about such an idea in A brief rant on the future of interaction design. In such a future, paper might well be dispensed with, but writing might in some ways return to more physicality than less. Even if not, we’re at a point where we can unlock unbridled creativity in children barely old enough to talk. It’s not time to slam on the brakes, but to go full-speed ahead and see where the journey takes us.

November 10, 2011. Read more in: Apple, Technology

4 Comments

Why can’t Apple’s notebook rivals innovate? Or: a rant about design rip-offs

Marco Arment on the Asus Zenbook (and with a less-than-subtle ‘the assbook air’ URL):

It’s sad, really, that the state-of-the-art in the PC world is attempting to copy Apple. Why isn’t Asus trying to blow the MacBook Air out of the water with something radically better?

Looking at the insides of the two devices, it’s almost criminal; it really looks like ASUS bought a MacBook Air, tore it down and told its engineers to reproduce it. The even more depressing thing: despite a bundle of cash from Intel and having Apple to use as a template, these other companies cannot match the MacBook Air. Every new ‘ultrabook’ that appears has some massive problem or other: a crappy screen, a rubbish trackpad, overheating. Of course, blogs are still banging on about the ‘Apple tax’, but when you’re paying over a grand for a notebook, would you really want to save a couple of hundred bucks by buying what almost amounts to counterfeit goods?

More to the point, this showcases problems in the tech industry as a whole. As Arment says, ASUS and other notebook makers shouldn’t be copying Apple—they should be trying to better it. And yet all we see in the market is Apple-like designs showing up a year after Apple’s released something, and often, comically, after it’s moved on to something new. Remember that rash of MacBook Pro clones when Apple unveiled the MacBook Air? Embarrassing.

The same’s true in the smartphone space. I’m hoping Samsung gets nailed to the wall worldwide in its legal spat with Apple, not because I dislike Android (despite what you may think from this blog, I don’t really care either way for the platform), but because I utterly despise the kind of lazy pilfering that goes on in the market these days. I’m sure some smart-arse will yell “XEROX! Ahuh-huh-huh” at me. Sure, because Mac OS was exactly like Xerox, and Xerox didn’t at all invite Apple over knowing full-well what it was developing and also get a ton of stock for good measure…

At any rate, Apple’s never really claimed to invent a great deal of things anyway. The company has at its best been about refinement, and its rivals never manage that. Had Asus come out with something that largely resembled the MacBook Air but somehow took it to the next level, that would have been close to the Xerox/Mac OS scenario, and that would also have been great. Something new. Something exciting. Something where you’d be saying: you know, Apple should really have created this. Instead, we just get knock-offs that do little more than dilute the original design and attempt to confuse people into buying something because it’s just like the (slightly) more expensive real deal, even though it’s clearly not.

November 10, 2011. Read more in: Apple, Technology

2 Comments

« older postsnewer posts »