Online retailers may be shooting themselves in the tail — the long tail, that is, according to Kartik Hosanagar, Wharton professor of operations and information management, and Dan Fleder, a Wharton doctoral candidate, in new research on the “recommenders” that many of these retailers use on their websites. Recommenders — perhaps the best known is Amazon’s — tend to drive consumers to concentrate their purchases among popular items rather than allow them to explore and buy whatever piques their curiosity, the two scholars suggest in a working paper titled, “Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity.”
Hosanagar and Fleder argue that online recommenders “reinforce the blockbuster nature of media.” And they warn that, by deploying standard designs, online retailers may be recreating the very phenomenon — circumscribed media purchasing choices — that some of them have bragged about helping consumers escape.
If you follow retailing, you know that “the long tail” is a phenomenon popularized by Chris Anderson, editor-in-chief of Wired magazine. The term, taken from statistics, points to the fact that online shoppers have shown themselves to be far more willing to purchase niche products — like documentary movies and old-time acoustic blues recordings — than marketers ever imagined. Outfits like Amazon and Apple’s iTunes figured this out because, with dramatically lower inventory costs than their bricks-and-mortar competitors, they could afford to carry huge lists of titles. Individually, many of those books or recordings didn’t sell often, but as a group, they outsold the hits on which media businesses had previously depended.
Anderson took the term — which, thanks to his book of the same name, became a marketing buzzword — from the graphical depiction of a statistical distribution. A typical distribution, when graphed, has a “head” — the fat, tall part — and a long “tail” that trailed off to the right. When online retailers graphed consumers’ purchases, they found that the distribution had a shorter head and a far longer tail than anyone would have predicted — i.e., people were buying lots of offbeat products and were far more eclectic in their tastes than expected. Many consumers were purchasing Britney Spears’ latest single, for example, but a surprisingly large number were also scooping up everything from, say, Bessie Smith’s 1920s blues recordings to the movies of modern Polish director Krzysztof Kieslowski.
Suggestions for the Shopping Cart
Online retailers, like their old economy counterparts, seek out ways to prompt consumers to buy more, and that’s where the recommenders that Hosanagar and Fleder studied come in. Amazon, for example, suggests additional books or CDs based on the ones that customers put in their shopping cart. It’s the online equivalent of the record store clerk who tells someone buying a Radiohead album that he also might like one by the Pixies. “Automated recommendations can be even more influential than human ones,” Hosanagar says.
But they are not without shortcomings, the two scholars point out. “Because common recommenders recommend products based on sales and [consumer] ratings, they cannot recommend products with limited historical data, even if they would be rated favorably,” they write. “This can create rich-get-richer effects for popular products and vice-versa for unpopular ones, which results in less diversity.”
Consider a well-known example that Anderson has used — Touching the Void, a book by mountaineer Joe Simpson about an Andean climb gone awry. Simpson’s book had nearly fallen out of print when Into Thin Air, John Krakauer’s bestseller about a disastrous day on Mt. Everest, became a bestseller. Amazon’s recommender suggested Touching the Void to buyers of Krakauer’s book, and Simpson’s title was reborn, seeing a surge in sales and eventually being made into a movie. Anderson heralds that as a victory of the long tail: Thanks to web retailing, consumers with a niche interest — mountaineering — found an obscure title that their local bookstores or libraries probably didn’t carry.
But Hosanagar and Fleder wonder if this example really points to the “sales concentration” that recommenders can create. “On their own, all of these readers of Into Thin Air might have bought a range of other books,” Hosanagar says. “But if most of them saw the same recommendation and bought Touching the Void, aggregate diversity was reduced.” Without the recommender, some shoppers might have found, say, Chris Bonnington’s tales of his climbing exploits in the Alps or Reinhold Messner’s account of his famed traverse of Nanga Parbat in the Pakistani Himalayas.
Recommenders stirred controversy in their early days, but not because of concerns about sales diversity. Web retailers feared that, by adding an extra step and a distraction, they might prompt consumers to abandon their purchases before they checked out. One of the developers of Amazon’s recommender, Greg Linden, was even told by his bosses to stop working on the project because they feared its effect on sales, according to a story recounted by Linden on his blog, Geeking with Greg. He therefore proposed testing the concept by letting some consumers see recommendations and others not. “The results were clear,” Linden writes. “Not only did [the recommender] win, but the feature won by such a wide margin that not having it live was costing Amazon a noticeable chunk of change. With new urgency, shopping cart recommendations launched.”
Linden, reached via email, declares himself untroubled by Hosanagar and Fleder’s findings. “Recommendation algorithms easily can be tuned to favor the back catalog — the long tail — as Netflix does,” he argues. Netflix, the online DVD purveyor, consciously highlights obscure titles in designing its recommender because they are less expensive for it to acquire and stock than well-known movies released by big studios. Netflix has said that its recommender increases not only the number of videos that customers rent, but also their loyalty to the company.
Linden also argues that, in the absence of online recommenders, consumers would turn to even cruder tools, like traditional bestseller lists. “You have to ask what content would otherwise be in place of the recommendations and whether that content would have greater diversity,” he says.
Hosanagar and Fleder agree. They are not disputing that recommenders beat old-school bestseller lists. Instead, they are saying that recommenders lead to less diversity in a world where consumers also have access to tools like search engines, blogs and social networks like MySpace and Facebook. “Relative to an ‘older’ world where choice was driven by bestseller lists, recommenders may increase diversity, but relative to a ‘newer’ world of search and many offerings, recommenders may decrease it,” they write.
Without recommenders, shoppers might instead turn to Amazon’s search engine or Yahoo and thus find one of Bonnington’s books instead of Simpson’s. Put differently, comparing today’s world without recommenders to one with only bestseller lists is like comparing a world without SUVs to one with only horse-drawn carriages. The meaningful comparison might be to a place with high-mileage hybrids.
Interestingly, from a single consumer’s point of view, recommenders can increase sales diversity, even as the overall diversity of sales in a given market decreases. “Recommenders can push each person to new products, but they often push us toward the same new products,” Hosanagar and Fleder write. Adds Fleder, “We’re discovering new items, but we’re all discovering the same items.”
At the very least, Hosanagar and Fleder want online retailers to understand the potential effects of recommenders that rely heavily on past sales. “I doubt that most of these firms have tried to study the diversity of their sales,” Hosanagar says. “I suspect that there’s a tacit assumption that recommenders drive diversity.”
The two scholars don’t expect online retailers to ditch recommenders. That would make little sense, given recommenders’ popularity with customers and the way that they boost sales. But firms might examine the assumptions behind the algorithms that power their recommenders and consider whether they are producing the desired results. If, for example, a firm believes that promoting sales diversity promotes its long-term interests, it might not want to rely too heavily on prior sales as a factor in its formula.
The Most Downloaded Application on Facebook
Many firms are devising innovative ways to generate recommendations that use other factors. In fact, a new online niche has emerged that consists of “firms whose business is to generate recommendations for you,” Fleder notes.
Consider iLike.com. “They capture every play from your iPod and every song in your iTunes and then make recommendations to you based on that,” he says. “There’s a social aspect as well, because it can also tell you what your friends are listening to,” Hosanagar adds. “It’s the most downloaded application on Facebook.” (To observe one’s friends’ listening habits, they have to sign up for iLike, too.)
Another example from the music business is Pandora, which bills itself as “free Internet radio” and grew out of the Music Genome Project, an encyclopedic effort to categorize music by genre, musician and mood. “They don’t recommend based on ratings but on the actual content,” Hosanagar says.
A question that arises out of Hosanagar and Fleder’s research is whether anyone besides unknown writers and unsigned musicians should care whether recommenders reinforce the blockbuster nature of the media business. As long as retailers are racking up sales and individual consumers are satisfied with their choices, does it matter that, in the absence of recommenders, consumers might opt for even greater diversity in their purchases?
The researchers point out that, from a consumer’s point of view, sales concentration involves tradeoffs. In theory, everyone’s tastes and needs differ, at least slightly, so more diversity can mean that consumers are finding more suitable products. But concentrated sales can have benefits, too. When people buy the same books and recordings as their friends, they can get enjoyment out of discussing them. That’s what happens when people band together in book clubs.
Such tradeoffs exist at the societal level as well, Hosanagar and Fleder say. Recommenders exemplify consumers’ growing power to focus on what they like and filter out all else. “This becomes telling when we consider that recommenders increasingly affect not only the music and movies we see but also, through customized newspapers, the news and political views we are exposed to,” says Fleder, adding that, as legal scholar Cass Sunstein notes, filtering and focusing on people’s interests creates a diverse society with many points of view, but commonality creates one whose members can understand each other.
Link to paper: Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity
