Ever since companies started recommending similar crap to stuff I’ve bought online at three in the morning on a drunken whim, I’ve become a whole lot poorer. But I’ve also laughed louder. Last year Amazon recommended I buy a nose hair trimmer because I’d purchased a book by Stephen Fry. Earlier this week, YouTube suggested I try Heroin.
Its rationale? I’d watched a video about Alice’s Adventures in Wonderland. No doubt YouTube found time to read Lewis Carroll’s work, to note the brazen references to experimental drugs and zoological hallucinations, and to suggest to all those viewing Wonderland-related miscellany that they might also enjoy hard drug how-tos, croquet flicks, and clips about missing cats.
It must take a lot of guts for big companies with fidgety legal departments to create a recommendation algorithm, feed it some visitor data, and sit back and hope for the best. Giving people the gift of discovery is a noble pursuit, but who could predict how disastrous the results can be?
You may also enjoy these illicit substances
To Amazon’s credit, they explain why they’ve picked items for you and allow you to forbid them from using certain past purchases to make future suggestions. By combining other users’ input, product ratings, and metadata about how items relate to each other, they’re building a collaborative filter that will improve matches for everyone over time; a sort of virtual collective consciousness for people with too much free time and disposable income, if you like.
If only YouTube offered the same Borg-like recommendation engine. I would like it to know that I don’t do hard drugs, and tell it that the Heroin video was a cheap stunt; a fun but harmless monologue only vaguely related to its title. (Of course I clicked it.) Devoid of any feedback system, the next time I visit YouTube it will surely upsell me to something harder and more expensive, unaware that it takes more than an ‘Eat Me’ tag to get me hooked.
Turns out that many of the companies who use recommendation engines know that there’s room for improvement, and that those leading the pack will pay handsomely for small performance gains. In 2006, Netflix offered a million dollars to anyone who could improve the accuracy — by just 10% — of computer-predicted ratings that users would give movies they hadn’t watched yet, which are based partly on how they rated other films.
The million dollar Netflix prize
The prize was awarded in 2009 to a team at AT&T Labs, one of whose paper entitled ‘The BellKor Solution to the Netflix Grand Prize’ [PDF] will give you a headache, which is probably why they published an article for laymen called The Million Dollar Programming Prize that’s well worth a read.
You’ll discover that the science of learning what people like is a complex and fascinating field. At its heart, a good recommendation engine is an artificially intelligent system; a moving model of irrational human tastes, built by a series of collaborative filters like ‘nearest-neighbours’, which intelligently discern related films and actors, and ‘latent factor models’, which help match users with films by classifying both into genres.
What fascinates me most is that the system not only works out, say, that you’ll probably like martial arts films or anything containing Keira Knightley; it also learns how critical you are as a human being. It then gives you a kind of secret ‘Scrooge rating’ so that it knows how to adjust other people’s scores to meet your overly high standards, and so that your unjust and frankly rude reviews are weighted appropriately when it makes suggestions to other users.
In the same way that you’ve accumulated enough information to know which of your friends and family would be delighted to watch Johnny Depp in the 2-hour drug-fuelled mindfuck that is Fear and Loathing in Las Vegas, and which of them would fail to see the funny side, so too is Netflix evolving to learn its users’ tastes. The only difference is that it doesn’t have to watch the films itself; it doesn’t need to — it forms its opinions by proxy based on what other people think, a surprisingly human trait.
All of which brings us loosely back to why YouTube recommends Heroin. Perhaps it’s not a true recommendation engine at all, but a random list of popular videos. Maybe it’s become so heavily gamed by its users that it’s now useless in determining our tastes. More likely, I suspect, that it gives crap suggestions because it’s never asked us what we like; it hasn’t gotten to know us like Netflix and Amazon have. When it comes to offering recommendations, YouTube is the sort of friend who defaults to chocs, socks, and smellies at Christmas every year because it’s given up figuring us out.
I think I know how it feels.
