The idea has gone from statistical curiosity to a deeply entrenched belief - especially in Silicon Valley: We are moving toward an economic state where it no longer becomes necessary to invest just in the top 1% of all ideas, but that instead an investor could create strategies using the web to take advantage of the statistical long tail - small niche markets (perhaps only one consumer deep) that in the aggregate can prove more lucrative than the blockbuster.
The Long Tail theory (which first appeared in a Wired article by Chris Anderson - later expanded to the book The Long Tail: Why the Future of Business is Selling Less of More in 2006 - has become in that interval nearly a mantra for small companies seeking venture capital or similar business funding. "Our strategy is to take advantage of the Long Tail by providing services to capture the broad range of niche markets through mass customization" has become almost boilerplate text in any high tech business proposal.
However, a recent report by Harvard Business School media analyst Anita Elberse has raised a number of significant questions about how viable this particular business model really is in a new web article Should You Invest in the Long Tail?. Anderson's basic thesis was that as the web has increased the availability of more obscure content (either that produced outside of traditional channels or that which was produced in the past and has largely disappeared from distribution) it has also made possible the notion that afficianados of particular genres, authors, musicians and so on could in fact take advantage of these new media and start consuming these products, giving them a second life and as such making them more profitable over time. His vision thus was that the long tail was becoming thicker over time, making blockbusters less important in the long scheme of things and ushering in an era where independent creators would reign supreme.
Not so fast, says Professor Elberse. In a six month study she evaluated sales figures from online music vendor Rhapsody, on-demand video vendor QuickFlix, and Nielsen VideoScan and SoundScan services, as well as performing less rigorous analyses on Amazon and other online "entertainment and publishing" vendors. What she found surprised her - far from becoming more robust, the evidence that she indicated that in fact the long tail is becoming both longer and flatter. In other words, rather than the more obscure content becoming more visible and affordable over time, what she discovered was that the blockbusters were in fact becoming more important to marketing, and more obscure content could in fact prove detrimental to effective sales because the cost per sale of each transaction actually goes up.
This, in fact, conforms remarkably well to William McPhee's Formal Theories of Mass Behavior, published in the early 1960s, which describe the traditional blockbuster pattern of marketing consumption, which argues that statistically, the top 1% of most large segmented markets (such as those found for entertainment products) typically does as well or better than the top 9% after it, and that there is a dramatic logarithmic falloff that makes the tail generally unprofitable.
One thing pointed out in the study was a curious pattern - Anderson described the connoisseur who would go and seek out the obscure titles in any given area, thus raising the profile of long tail purchases. However, Elberse sees an alternative pattern developing - that connoisseur is far more likely than the average to consume a significantly higher number of titles, but in general for every obscure title they choose, they are likely to choose two or more popular titles. The non-connoisseur, on the other hand, is far more likely to choose the popular titles almost exclusively. This has the net effect not of boosting long-tail purchasing, but in fact of suppressing it.
Andersen and Elberse have carried this debate online, with questions being raised of differences in methodology and potentially of interpretation, but this debate carries over into the world of software development as well. Do the same rules and patterns apply?
Open source software tends to follow the same kind of logarithm decay rules that media in general apply, and one of the more vocal (albeit not necessarily well-reasoned) arguments for the adoption of open source software is that there is benefit for those people working in the long tail part of a software niche distribution by virtue of the breadth of "similar" adoption.
However, for software developers there is an investment (in time, tools, and focus) that's made when any piece of software is used, and if the benefits of the long-tail really do not apply (the tail becomes longer and flatter over time rather than shorter and fatter) then it is worth it for developers to examine their choices heavily if they had assumed Anderson's thesis.
On the other hand, it's worth noting that there are other relationships at play too - early adopters are almost always working with long-tail technologies at first by definition - they are too new to have achieved broad exposure in the market. As with nearly any economic decision (and the choice of software is ultimately an economic decision) there are typically multiple variables at work, including, most importantly time, and as a consequence, the long tail description is only a snapshot of a considerably more complex multivariate function that says that while the top 1% of a market becomes perhaps more significant over time, the specific constituents of that top 1% are in constant flux and follow power laws of their own (such as the illustration showcasing the trajectory of a given "product" up and down the long tail over time).
Thus, before necessarily throwing out (or embracing) long tail dynamics based upon the Anderson vs. Elberse debates, it is worth understanding that there is more to the story than a simple two dimensional graph. Popularity is a remarkably transienty property ... the Romans stated this as the phrase "Sic transit gloria mundi", but which today the same sentiment may best be expressed as "today at the head of the curve, tomorrow somewhere out there on the long, long tail".