Less popular choices must give advantages to compensate for their unpopularity, but that doesn't mean they are "better." Many a small religious sect is bound together all the stronger for being a persecuted minority, and that bond may well the be advantage they seek.

Example: Cognitive therapy (vs. SSRIs like Prozac) - studies show that good cognitive therapy can bring about similar changes in brain patterns, and relapse is lower once treatment (or medication) is terminated. I don't know how expensive good therapy is (so the user-end costs might be the same), but Pfizer sure isn't making its buck on cognitive therapy. (You can google stats on SSRI use and comparative studies.)

Tons.

Web browsers, Operating Systems, analog videotape formats (Beta/VHS) (I'd look for this effect in hidef videodiscs soon as well), peripheral interconnects (USB/firewire), mobile phone transmission protocols (CDMA/GSM), mobile digital audio players (iPod/Zune/etc), programming languages (C++/Java/Python)...

Some examples here might be better than others.

If you can survive by being popular or superior, how does one get to be popular in the first place? I would think a lot of the times something is more popular because it is in fact superior in some way.

I remember long ago I had a 3do video game console in a time that most people still had SNES or were getting Playstations. I always consoled myself that I had the superior, though far less popular system. But the reality was I had the inferior product, and the popular product was so simply because it was better.

All products have tradeoffs, and for different consumers, the tradeoffs will vary. Some people prefer the most popular web browser because it is the one which will work at the greatest number of web sites. Others prefer a less popular browser because it has features the popular one does not. This allows the less popular browser to continue to exist even though the most popular one is better in some ways. The fact that multiple products exist doesn't prove that the less popular ones are better in general, merely that they are better for some people.

What about time and date formats? While some formats based on a single scalar value (e.g. Unixtime) are common in certain applications, the system most commonly used by humans to specify the time of day uses at least three different units (hours, minutes and seconds), with a conversion factor (60) that isn't a power of ten. The rules for calculating dates are even more complicated. Time zones complicate matters further; a single '%Y-%m-%d %H:%M:%S' string doesn't even unambiguously specify a point in time, unless one already knows what time zone the sender is using. From a purely algorithmical perspective, this is a really poor way of specifying a scalar (approximation; none of the systems mentioned deal with time dilation) value. While it encodes certain additional information (position of a planet relative to the local sun), it also makes performing arithmetic on datetimes a lot more difficult than it needs to be.

Hmmm... this fits with my theory on why English units still survive in a world with the metric system. (Which the time question makes me think of.) Which is to say, it seems to me that the metric system is vastly more convenient for doing math across different units -- not surprisingly, as the system is optimized for that. But in general, the units in English measure are more conveniently sized for everyday use. (Take temperature, for instance -- 0F to 100F is nearly perfect for everyday temperatures, at least here in Michigan -- temperatures outside that range are rare events (just a couple days a year, usually), whereas temperatures from 0-10 and 90-100 are much more common.)

The fun thing here is you can make the argument both ways depending on the scale of your comparison -- either the metric system survives in the scientific community in the US because it is superior for scientific matters, or the English system survives in the US (against the world) because it is superior in everyday use.

Forgive me - I'm not a statistician nor an economist - but isn't this just a pareto distribution thing?

In the comments section of this post http://www.overcomingbias.com/2006/12/bosses_prefer_o.html http://www.overcomingbias.com/2006/12/bosses_prefer_o.html#comments, Perry provided what seemed to me be a very insightful analysis of how organizations, as they scale, switch from being entrepreneurial and innovative to sclerotic and less competitive, largely through the arrival of what I would call "cashflow appropriators", who as Perry says drive out those with real competence forcing them into more entrepreneurial situations. In the market for goods and services (and even education), reputation and cashflows are much more durable than people credit, and can mask incompetence for, well..., longer than you (the innovative competitor) can remain solvent. That means the products that dominate may very often be technically or functionally inferior.

Political managers know they can get away with this (intuitively, or consciously and cynically), and care less for the innovation process, than how to grab the cashflow once created. They avoid the tricky bit, which in the old days literally involved getting your hands dirty, but now probably means knowing some "code". These people have more time on their hands to cultivate their reputations, and so will be more visible to us through the press. They are generally less interesting than their PR, of which business schools can form an important location of support. In truth they are not news, or really newsworthy.

Innovators, where they see an opportunity, will try and attack those markets/companies where there is a chance of undermining the dominant player, and diverting those cashflows. There is no guarantee that a superior product will succeed. There is a lot of necessary delusion and/or stoicism among entrepreneurs, artists and even sportsmen to reach the top, which is overlooked in the backstory. Product innovation may have to go through several cycles and different companies to take off, where each iteration may already be superior to the incumbent, but not good enough or lucky enough to break through.

Then, the last company/product succeeds on the back of the earlier partial innovations. Just like the last person to take the lid of the marmalade jar, the winner attributes success to his own strength rather than the cumulative small, frictional movements of his predecessors. The Harvard case study is written in his favour, and this folklore eventually will be discussed ad nauseum in the tutorial, boardroom, at the watercooler, and in the blogosphere to bolster all sorts of spurious arguments and subsequent copycat strategies, which don't succeed because all the data for success has not been captured.

Of course, functional and technical superiority may be too narrow a definition of the "product". To be fair to incumbents, a product may be superior because the information on reliability that the brand carries, or the service or retail network, reduces the information costs of assessing the alternatives for some significant number of users. This explains some of the attitude of the appropriator type in pursuing a defensive satisficing strategy. Also, on the part of the consumer (and I am like this with PCs), sunk cost biases can also be quite durable. A lot of information may need to reach me before I see the value of switching. The "product guy", a persona we all seem to adopt in these kinds of conversation, underestimates this at his peril.

Academically, I'm not sure how to apply this. But could one propose that neo-classical economics has a lot of vested interest built into it? Behavioural economics, which I hear requires a smaller body of literature to master and may be more applicable/superior to real world marketing and political issues, would require a lot of academics to abandon a lifetime's work, which they are of course not inclined to do. But at some point, this situation will tip.

I think the English measurement story is simply one of path dependence. It is entrenched, lots of people know it, and it would cost a lot in infrastructure and learning to switch, just like the QWERTY keyboard. OTOH, the English language has considerable nuance, given the many languages that go into it.

The Pareto distribution argument is in the right direction. Think of a skewed distribution versus a normal one. So, the mean of the normal one might be higher than that of the skewed one. On average it does better, and hence may be more popular. But the skewed one has this tail that does much better than the normal one a non-trivial amount of the time, so that risk lovers are attracted to it. This is not all that different from the argument about how noise traders survive in financial markets. Most go bankrupt, but those who actually did buy low and sell high do better than anybody else in the market and definitely survive.

You survive by surviving. No shortage of ways to achieve it.

the argument about how noise traders survive

Surely the argument you give--that false beliefs can lead to extra risk, increasing expected returns while decreasing expected utility--is older than the noise trader literature?

Douglas Knight,

There is an older literature for sure, but it was largely dismissed during the heyday of the rational expectations revolution in the 1970s and 1980s. The first break was the presidential speech to the American Finance Association in 1985 by Fischer Black, pubbed in 1986 in the Journal of Finance as "Noise," followed by the 1987 stock market crash, and then a pair of rigorous papers in 1990 and 1991 in the QJE and Journal of Business by the formidable quartet of Bradford DeLong, Andrei Shleifer, Lawrence Summers, and Robert Waldmann. They in particular argued that a rational trader must take into account the beliefs and actions of the noise traders, which can then get one into the realm of the old problem, going back to Keynes, of self-fulfilling prophecies. The issue is not false beliefs, it is understanding that the market may be driven by people who are not following long-run fundamentals and taking accurate advantage of their behavior. It is a matter of accurately forecasting a market, in the case of a bubble, accurately forecasting the path of the bubble, and those who do so can and will make more money than the safe, fundamentalist trader who stays away from the bubble.

Barkley Rosser

How good are the bubble forecasters, and can these phenomenon be usefully measured quantitatively? My impression from Mandelbrot's book was not perfectly yet. I've seen housing market analysis from people like Didier Sornette, who would imply we should have experienced a crash in the UK before now, although there may be special factors here since his analysis like large-scale immigration that have contributed to a soft landing. Are you thinking of the more intuitive, but post-rationalising behaviour of a Soros, or the very calculating activities of a Taleb, but who himself seems to value intuition? How large is the population of rational traders? My guess is not that large, and they too must be quite vulnerable to failure and exit from the market, because of the loss aversion of their employers, as Taleb describes.

knackeredhack,

Forecasting markets in general is a mug's game, only made worse when bubbles go on. Go read all the debates about the (maybe) housing bubble going on over at Econbrowser. There is a deep argument, which Jim Hamilton has long been one of the main promulgators of, that one can never identify for certain econometrically if one is in a bubble or not, although one may be able to do so pretty much for certain sometimes with closed-end funds, where there is a well-defined fundamental in the net asset value of the fund.

Regarding Sornette, his model is one of a rational stochastically crashing bubble, which requires a sharp upward acceleration to provide risk premia for the rising probability of the inevitable crash. These tend to go to infinity at a certain point, which is the basis for forecasting the crash that Sornette uses. Of course there is plenty of reason to believe that people in bubbles are not fully rational, and therefore it is not surprising that Sornette has had a rather mixed record in his forecasting.

Well, interesting topic! But I don't agree that the frequentist view is the worst. Seems to me as the most objective, the most "scientific" one. The one that can best explain the concept of probability. The subjective view doesn't explain much to me, and the propensity just doesn't seem right, because I don't believe in probabilities being inherent to anything, they're can be volatile under a single object.

Or maybe I just might read more about this whole subject, since my opinions come solely from this article.

If someone disagrees, I'd like to know why. Good to know different points of view =)

I think there's substantial overlap between some of your ideas here and Richard Gabriel's famous "Worse is Better". He wrote several follow-on essays, too, all linked to from there. You might want to have a look at them.

Learning and memorization.

Spaced repetition, the testing effect and the use of mnemonics has not replaced linear non tested study (where you read something over multiple times in a short period of time, don't test yourself and don't space the readings.)

Isn't this just a special case of Berkson's paradox?

Reverse stupidity is not wisdom. Here we have reversed ad populus (aka The Hipster's Fallacy). Pepsi and Macs are not strictly superior to their more popular counterparts by dent of existing. Rather, their existence is explained by comparative advantage in some cases for some users.