Forgive me - I'm not a statistician nor an economist - but isn't this just a pareto distribution thing?
In the comments section of this post http://www.overcomingbias.com/2006/12/bosses_prefer_o.html http://www.overcomingbias.com/2006/12/bosses_prefer_o.html#comments, Perry provided what seemed to me be a very insightful analysis of how organizations, as they scale, switch from being entrepreneurial and innovative to sclerotic and less competitive, largely through the arrival of what I would call "cashflow appropriators", who as Perry says drive out those with real competence forcing them into more entrepreneurial situations. In the market for goods and services (and even education), reputation and cashflows are much more durable than people credit, and can mask incompetence for, well..., longer than you (the innovative competitor) can remain solvent. That means the products that dominate may very often be technically or functionally inferior.
Political managers know they can get away with this (intuitively, or consciously and cynically), and care less for the innovation process, than how to grab the cashflow once created. They avoid the tricky bit, which in the old days literally involved getting your hands dirty, but now probably means knowing some "code". These people have more time on their hands to cultivate their reputations, and so will be more visible to us through the press. They are generally less interesting than their PR, of which business schools can form an important location of support. In truth they are not news, or really newsworthy.
Innovators, where they see an opportunity, will try and attack those markets/companies where there is a chance of undermining the dominant player, and diverting those cashflows. There is no guarantee that a superior product will succeed. There is a lot of necessary delusion and/or stoicism among entrepreneurs, artists and even sportsmen to reach the top, which is overlooked in the backstory. Product innovation may have to go through several cycles and different companies to take off, where each iteration may already be superior to the incumbent, but not good enough or lucky enough to break through.
Then, the last company/product succeeds on the back of the earlier partial innovations. Just like the last person to take the lid of the marmalade jar, the winner attributes success to his own strength rather than the cumulative small, frictional movements of his predecessors. The Harvard case study is written in his favour, and this folklore eventually will be discussed ad nauseum in the tutorial, boardroom, at the watercooler, and in the blogosphere to bolster all sorts of spurious arguments and subsequent copycat strategies, which don't succeed because all the data for success has not been captured.
Of course, functional and technical superiority may be too narrow a definition of the "product". To be fair to incumbents, a product may be superior because the information on reliability that the brand carries, or the service or retail network, reduces the information costs of assessing the alternatives for some significant number of users. This explains some of the attitude of the appropriator type in pursuing a defensive satisficing strategy. Also, on the part of the consumer (and I am like this with PCs), sunk cost biases can also be quite durable. A lot of information may need to reach me before I see the value of switching. The "product guy", a persona we all seem to adopt in these kinds of conversation, underestimates this at his peril.
Academically, I'm not sure how to apply this. But could one propose that neo-classical economics has a lot of vested interest built into it? Behavioural economics, which I hear requires a smaller body of literature to master and may be more applicable/superior to real world marketing and political issues, would require a lot of academics to abandon a lifetime's work, which they are of course not inclined to do. But at some point, this situation will tip.
Barkley Rosser
How good are the bubble forecasters, and can these phenomenon be usefully measured quantitatively? My impression from Mandelbrot's book was not perfectly yet. I've seen housing market analysis from people like Didier Sornette, who would imply we should have experienced a crash in the UK before now, although there may be special factors here since his analysis like large-scale immigration that have contributed to a soft landing. Are you thinking of the more intuitive, but post-rationalising behaviour of a Soros, or the very calculating activities of a Taleb, but who himself seems to value intuition? How large is the population of rational traders? My guess is not that large, and they too must be quite vulnerable to failure and exit from the market, because of the loss aversion of their employers, as Taleb describes.