Statistical Prediction Rules Out-Perform Expert Human Judgments

68 lukeprog 18 January 2011 03:19AM

A parole board considers the release of a prisoner: Will he be violent again? A hiring officer considers a job candidate: Will she be a valuable asset to the company? A young couple considers marriage: Will they have a happy marriage?

The cached wisdom for making such high-stakes predictions is to have experts gather as much evidence as possible, weigh this evidence, and make a judgment. But 60 years of research has shown that in hundreds of cases, a simple formula called a statistical prediction rule (SPR) makes better predictions than leading experts do. Or, more exactly:

When based on the same evidence, the predictions of SPRs are at least as reliable as, and are typically more reliable than, the predictions of human experts for problems of social prediction.1

For example, one SPR developed in 1995 predicts the price of mature Bordeaux red wines at auction better than expert wine tasters do. Reaction from the wine-tasting industry to such wine-predicting SPRs has been "somewhere between violent and hysterical."

How does the SPR work? This particular SPR is called a proper linear model, which has the form:

P = w1(c1) + w2(c2) + w3(c3) + ...wn(cn)

The model calculates the summed result P, which aims to predict a target property such as wine price, on the basis of a series of cues. Above, cn is the value of the nth cue, and wn is the weight assigned to the nth cue.2

In the wine-predicting SPR, c1 reflects the age of the vintage, and other cues reflect relevant climatic features where the grapes were grown. The weights for the cues were assigned on the basis of a comparison of these cues to a large set of data on past market prices for mature Bordeaux wines.3

There are other ways to construct SPRs, but rather than survey these details, I will instead survey the incredible success of SPRs.

continue reading »

Buy Insurance -- Bet Against Yourself

29 MBlume 26 November 2010 04:48AM
My friend and housemate, User:Kevin, makes a very pleasant living selling opioids on the internet, a living he expects to continue for some time, unless something awful happens like Obama losing the next election. The Intrade contract for Obama's loss is currently trading at 42% -- what can User:Kevin do about this?
My suggestion was that he bet heavily on Obama's loss. Say he spends $4200 buying not-Obama futures. If Obama wins, that money becomes worthless, but he gets four years selling kratom regulation-free. On the other hand, say Palin takes a surprise victory and institutes draconian regulation on various substances -- User:Kevin's $4,200 has just become $10,000, leaving a $5800 windfall to help him while he finds his next muse.
This is nothing more than what we normally call buying insurance, just extended to whatever outcomes you may want to insure against. Let's talk about some of the effects of this action.
continue reading »

The hard limits of hard nanotech

19 lsparrish 07 November 2010 12:49AM

What are the plausible scientific limits of molecular nanotechnology?

Richard Jones, author of Soft Machines has written an interesting critique of the room-temperature molecular nanomachinery propounded by Drexler:

Rupturing The Nanotech Rapture

If biology can produce a sophisticated nanotechnology based on soft materials like proteins and lipids, singularitarian thinking goes, then how much more powerful our synthetic nanotechnology would be if we could use strong, stiff materials, like diamond. And if biology can produce working motors and assemblers using just the random selections of Darwinian evolution, how much more powerful the devices could be if they were rationally designed using all the insights we've learned from macroscopic engineering.

But that reasoning fails to take into account the physical environment in which cell biology takes place, which has nothing in common with the macroscopic world of bridges, engines, and transmissions. In the domain of the cell, water behaves like thick molasses, not the free-flowing liquid that we are familiar with. This is a world dominated by the fluctuations of constant Brownian motion, in which components are ceaselessly bombarded by fast-moving water molecules and flex and stretch randomly. The van der Waals force, which attracts molecules to one another, dominates, causing things in close proximity to stick together. Clingiest of all are protein molecules, whose stickiness underlies a number of undesirable phenomena, such as the rejection of medical implants. What's to protect a nanobot assailed by particles glomming onto its surface and clogging up its gears?

The watery nanoscale environment of cell biology seems so hostile to engineering that the fact that biology works at all is almost hard to believe. But biology does work--and very well at that. The lack of rigidity, excessive stickiness, and constant random motion may seem like huge obstacles to be worked around, but biology is aided by its own design principles, which have evolved over billions of years to exploit those characteristics. That brutal combination of strong surface forces and random Brownian motion in fact propels the self-assembly of sophisticated structures, such as the sculpting of intricately folded protein molecules. The cellular environment that at first seems annoying--filled with squishy objects and the chaotic banging around of particles--is essential in the operation of molecular motors, where a change in a protein molecule's shape provides the power stroke to convert chemical energy to mechanical energy.

In the end, rather than ratifying the ”hard” nanomachine paradigm, cellular biology casts doubt on it. But even if that mechanical-engineering approach were to work in the body, there are several issues that, in my view, have been seriously underestimated by its proponents.

...

Put all these complications together and what they suggest, to me, is that the range of environments in which rigid nanomachines could operate, if they operate at all, would be quite limited. If, for example, such devices can function only at low temperatures and in a vacuum, their impact and economic importance would be virtually nil.

The entire article is definitely worth a read. Jones advocates more attention to "soft" nanotech, which is nanomachinery with similar design principles to biology -- the biomimetic approach -- as the most plausible means of making progress in nanotech.

As far as near-term room-temperature innovations, he seems to make a compelling case. However the claim that "If ... such devices can function only at low temperatures and in a vacuum, their impact and economic importance would be virtually nil" strikes me as questionable. It seems to me that atomic-precision nanotech could be used to create hard vacuums and more perfectly reflective surfaces, and hence bring the costs of cryogenics down considerably. Desktop factories using these conditions could still be feasible.

Furthermore, it bears mentioning that cryonics patients could still benefit from molecular machinery subject to such limitations, even if the machinery is not functional at anything remotely close to human body temperature. The necessity of a complete cellular-level rebuild is not a good excuse not to cryopreserve. As long as this kind of rebuild technology is physically plausible, there arguably remains an ethical imperative to cryopreserve patients facing the imminent prospect of decay.

In fact, this proposed limitation could hint at an alternative use for cryosuspension that is entirely separate from its present role as an ambulance to the future. It could perhaps turn out that there are forms of cellular surgery and repair which are only feasible at those temperatures, which are nonetheless necessary to combat aging and its complications. The people of the future might actually need to undergo routine periods of cryogenic nanosurgery in order to achieve robust rejuvenation. This would be a more pleasant prospect than cryonics in that it would be a proven technology at that point; and most likely the vitrification process could be improved sufficiently via soft nanotech to reduce the damage from cooling itself significantly.

Bayesian Collaborative Filtering

14 JGWeissman 03 April 2010 11:29PM

I present an algorithm I designed to predict which position a person would report for an issue on TakeOnIt, through Bayesian updates on the evidence of other people's positions on that issue. Additionally, I will point out some potential areas of improvement, in the hopes of inspiring others here to expand on this method.


For those not familiar with TakeOnIt, the basic idea is that there are issues, represented by yes/no questions, on which people can take the positions Agree (A), Mostly Agree (MA), Neutral (N), Mostly Disagree (MD), or Disagree (D). (There are two types of people tracked by TakeOnIt: users who register their own opinions, and Experts/Influencers whose opinions are derived from public quotations.)

The goal is to predict what issue a person S would take on a position, based on the positions registered by other people on that question. To do this, we will use Bayes' Theorem to update the probability that person S takes the position X on issue I, given that person T has taken position Y on issue I:

P(S takes X on I | T takes Y on I) = P(S takes X on I)*P(T takes Y on I | S takes X on I)/P(T takes Y on I)

Really, we will be updating on several people Tj taking positions Ty on I:

P(S takes X on I | for all j, Tj takes Yj on I) = P(S takes X on I)*Product over j of (P(Tj takes Yj on I | S takes X on I)/P(Tj takes Yj on I))

continue reading »

Creating a Less Wrong prediction market

6 Kevin 26 February 2010 11:48AM

I will bet 500 karma that a funny picture thread will appear on Less Wrong within one year. If anyone is interested in the bet, we can better define terms.

Right now the LW software doesn't support karma transfers. Until it does and we can develop a more robust prediction market, let's just record the karma transfers on the wiki page that already exists for this purpose.

I will also give 100 karma to anyone that donates $10 to the SIAI before the current fundraising campaign is over.

10,000 karma for the first person with a karma transfer source code patch?

The Prediction Hierarchy

21 RobinZ 19 January 2010 03:36AM

Related: Advancing Certainty, Reversed Stupidity Is Not Intelligence

The substance of this post is derived from a conversation in the comment thread which I have decided to promote. Teal;deer: if you have to rely on a calculation you may have gotten wrong for your prediction, your expectation for the case when your calculation is wrong should use a simpler calculation, such as reference class forecasting.

Edit 2010-01-19: Toby Ord mentions in the comments Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes (PDF) by Toby Ord, Rafaela Hillerbrand, and Anders Sandberg of the Future of Humanity Institute, University of Oxford. It uses a similar mathematical argument, but is much more substantive than this.

A lottery has a jackpot of a million dollars. A ticket costs one dollar. Odds of a given ticket winning are approximately one in forty million. If your utility is linear in dollars, should you bet?

The obvious (and correct) answer is "no". The clever (and incorrect) answer is "yes", as follows:

According to your calculations, "this ticket will not win the lottery" is true with probability 99.9999975%. But can you really be sure that you can calculate anything to that good odds? Surely you couldn't expect to make forty million predictions of which you were that confident and only be wrong once. Rationally, you ought to ascribe a lower confidence to the statement: 99.99%, for example. But this means a 0.01% chance of winning the lottery, corresponding to an expected value of a hundred dollars. Therefore, you should buy the ticket.

The logic is not obviously wrong, but where is the error?

continue reading »

The New Nostradamus

13 Kaj_Sotala 12 September 2009 02:42PM

I stumbled upon an article called The New Nostradamus, reporting of a game-theoretic model which predicts political outcomes with startling effectiveness. The results are very impressive. However, the site hosting the article is unfamiliar to me, so I'm not certain of the article's verity, but a quick Google seems to support the claims, at least on a superficial skimming. Here's his TED talk. The model seems almost too good to be true, though. Anybody know more?

Some choice bits from the article:

The claim:

In fact, the professor says that a computer model he built and has perfected over the last 25 years can predict the outcome of virtually any international conflict, provided the basic input is accurate. What’s more, his predictions are alarmingly specific. His fans include at least one current presidential hopeful, a gaggle of Fortune 500 companies, the CIA, and the Department of Defense.

The results:

The criticism rankles him, because, to his mind, the proof is right there on the page. “I’ve published a lot of forecasting papers over the years,” he says. “Papers that are about things that had not yet happened when the paper was published but would happen within some reasonable amount of time. There’s a track record that I can point to.” And indeed there is. Bueno de Mesquita has made a slew of uncannily accurate predictions—more than 2,000, on subjects ranging from the terrorist threat to America to the peace process in Northern Ireland—that would seem to prove him right.

[...]

To verify the accuracy of his model, the CIA set up a kind of forecasting face-off that pit predictions from his model against those of Langley’s more traditional in-house intelligence analysts and area specialists. “We tested Bueno de Mesquita’s model on scores of issues that were conducted in real time—that is, the forecasts were made before the events actually happened,” says Stanley Feder, a former high-level CIA analyst. “We found the model to be accurate 90 percent of the time,” he wrote. Another study evaluating Bueno de Mesquita’s real-time forecasts of 21 policy decisions in the European community concluded that “the probability that the predicted outcome was what indeed occurred was an astounding 97 percent.” What’s more, Bueno de Mesquita’s forecasts were much more detailed than those of the more traditional analysts. “The real issue is the specificity of the accuracy,” says Feder. “We found that DI (Directorate of National Intelligence) analyses, even when they were right, were vague compared to the model’s forecasts. To use an archery metaphor, if you hit the target, that’s great. But if you hit the bull’s eye—that’s amazing."

continue reading »

Kling, Probability, and Economics

1 matt 30 March 2009 05:15AM

Related to: Beautiful Probability, Probability is in the Mind

Arnold Kling ponders probability:

How one thinks about probability affects how one thinks about economics. Consider the use of the word "probability" in each of the following sentences:

1. What is the probability that when a fair coin is flipped it will come up heads?
2. What is the probability that exactly two number-one seeds will make it to the final four in the March Madness basketball tournament?
3. What is the probability that New York City will rank higher relative to other cities five years from now in terms of college graduates?

We would answer the first question by saying that the probability is 50 percent, based on the very definition of a fair coin. This is an axiomatic interpretation of probability. The axiomatic view treats probability as a matter of pure logic, with statements that do not require any empirical testing.

We would answer the second question by looking up historical records for the NCAA basketball tournament. This is the frequentist account of probability, which treats probability as counting outcomes from repeated trials. A frequentist would claim that the only way we can know that a coin has a 50 percent probability of coming up heads is by actually flipping a coin enough times to verify this empirically.

The third question cannot be answered on the basis of axioms or observed frequencies. The probability estimate is purely subjective. The subjective account of probability is that it reflects an individual belief that cannot be proven either logically or empirically.

In the tradition of Reddit, and a little inspired by Robin, this is a simple link to an interesting page somewhere else - I leave comment and discussion to the very awesome Less Wrong community.

Edit: Eliezer has in the past been uncomplimentary of the "accursèd frequentists". In at least Beautiful Probability and Probability is in the Mind, he has characterized (for at least some problems) the "frequentist" approach as being wrong, and the "Bayesian" approach as being right. Kling suggests different problems for which different approaches are approrpriate.

Individual Rationality Is a Matter of Life and Death

24 patrissimo 21 March 2009 07:22PM

On at least two occasions - one only a year past - my life was at serious risk because I was not thinking clearly.  Both times, I was lucky (and once, the car even survived!).  As a gambler I don't like counting on luck, and I'd much rather be rational enough to avoid serious mistakes.  So when I checked the top-ranked posts here and saw Robin's Rational Me or We? arguing against rationality as a martial art I was dumbfounded.  To me, individual rationality is a matter of life and death[1].

In poker, much attention is given to the sexy art of reading your opponent, but the true veteran knows that far more important is the art of reading and controlling yourself.  It is very rare that a situation comes up where a "tell" matters, and each of my opponents is only in an occasional hand.  I and my irrationalities, however, are in every decision in every hand.  This is why self-knowledge and self-discipline are first-order concerns in poker, while opponent reading is second or perhaps even third.

And this is why Robin's post is so wrong[2].  Our minds and their irrationalities are part of every second of our lives, every moment we experience, and every decision that we make.  And contra to Robin's security metaphor, few of our decisions can be outsourced.  My two bad decisions regarding motor vehicles, for example, could not have easily been outsourced to a group rationality mechanism[3].  Only a tiny percentage of the choices I make every day can be punted to experts.

continue reading »

Rational Me or We?

116 RobinHanson 17 March 2009 01:39PM

Martial arts can be a good training to ensure your personal security, if you assume the worst about your tools and environment.  If you expect to find yourself unarmed in a dark alley, or fighting hand to hand in a war, it makes sense.  But most people do a lot better at ensuring their personal security by coordinating to live in peaceful societies and neighborhoods; they pay someone else to learn martial arts.  Similarly, while "survivalists" plan and train to stay warm, dry, and fed given worst case assumptions about the world around them, most people achieve these goals by participating in a modern economy.

The martial arts metaphor for rationality training seems popular at this website, and most discussions here about how to believe the truth seem to assume an environmental worst case: how to figure out everything for yourself given fixed info and assuming the worst about other folks.  In this context, a good rationality test is a publicly-visible personal test, applied to your personal beliefs when you are isolated from others' assistance and info.  

I'm much more interested in how we can can join together to believe truth, and it actually seems easier to design institutions which achieve this end than to design institutions to test individual isolated general tendencies to discern truth.  For example, with subsidized prediction markets, we can each specialize on the topics where we contribute best, relying on market consensus on all other topics.  We don't each need to train to identify and fix each possible kind of bias; each bias can instead have specialists who look for where that bias appears and then correct it. 

Perhaps martial-art-style rationality makes sense for isolated survivalist Einsteins forced by humanity's vast stunning cluelessness to single-handedly block the coming robot rampage.  But for those of us who respect the opinions of enough others to want to work with them to find truth, it makes more sense to design and field institutions which give each person better incentives to update a common consensus.

View more: Prev | Next