1001 PredictionBook Nights
I explain what I've learned from creating and judging thousands of predictions on personal and real-world matters: the challenges of maintenance, the limitations of prediction markets, the interesting applications to my other essays, skepticism about pundits and unreflective persons' opinions, my own biases like optimism & planning fallacy, 3 very useful heuristics/approaches, and the costs of these activities in general.
Plus an extremely geeky parody of Fate/Stay Night.
This essay exists as a large section of my page on predictions markets on gwern.net: http://www.gwern.net/Prediction%20markets#1001-predictionbook-nights
Rationality Power Tools
Summary: Rationalists should win; however, it could take a really long time before a technological singularity or uploading provide powerful technology to aid rationalists in achieving their goals. It's possible today to create assistant computer software to help direct human effort and provide "hints" for clearer thinking. We should catalog such software when it exists and create it when it doesn't.
The Problem
We may be waiting awhile for a Friendly AI or similar “world changing” technology to appear. While technology continues to improve, the process of creating a Friendly AI seems extremely tricky, and there’s no solid ETA on the program. Uploading is still years to decades away. In the meantime, aspiring rationalists still have to get on with our lives.
Rationality is hard. Merely knowing about a bias is often not enough to overcome it. Even in cases where the steps to act rationally are known, the algorithm required may be more than can be done manually, or may require information which itself is not immediately at hand. However, a lot of things that are difficult become easier when you have the right tools. Could there be tools that supplement the effort involved in making a good decision? I suspect that this is the case, and will give several examples of programs that the community could work to create -- computer software to help you win. Because a lot of software is specifically created to address problems as they come up, it would also be worthwhile to maintain an index of already available software with special usefulness and applicability to Less Wrong readers.
Hayekian Prediction Markets?
I think I basically get the idea behind prediction markets. People take their money seriously, so the opinions of people who are confident enough to bet real money on those opinions deserve to be taken seriously as well. That kid on the schoolyard who was always saying "wanna bet?" might have been annoying but he also had a point: your willingness or unwillingness to bet does say something about how seriously your opinions ought to be taken. Furthermore, there are serious problems with the main alternative prediction method, which consists of asking experts what they think is going to happen. Almost nobody ever keeps track of whose predictions turned out to be right and then listens to those people more. Some predictions involve events that are so rare or so far in the future that there's no way for an expert to accumulate a track record at all. Some issues give experts incentives to be impressively wrong rather than boringly right. And so on. These are all good points, and they make enough sense to me to convince me that prediction markets deserve to be taken seriously and tested empirically. If they reliably produce better predictions than the alternatives, then they deserve to win the day.*
But there is a particular claim that is made about prediction markets that I am skeptical of. It starts with the well-known idea, usually associated with Friedrich on Hayek, that a major virtue of free markets is that there is all kinds of useful information spread out in local chunks throughout the economy, which individuals can usefully exploit but a central planner never could, which is reflected in market prices, and which in turn cause resources to be allocated efficiently. It then goes on to argue that prediction markets have a similar virtue. As an example, suppose there's a prediction market for a national election, and you happen to know that Candidate X is more popular in your little town than most people think. There's no way that some faraway expert could have known this or incorporated it into his or her prediction in any way, but it gives you an incentive to bet on Candidate X, which causes your local information to be reflected in the prediction market price. Lots and lots of people doing the same thing will cause lots and lots of such little local pieces of information, which couldn't have been obtained any other way, to also be reflected in the market price.
PredictionBook.com - Track your calibration
Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.
The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.
I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)
Wrong Tomorrow
Wrong Tomorrow by Maciej Cegłowski is a very simple site for listing pundit predictions and tracking them [FAQ]. It doesn't come with prices and active betting... but a simple registry of this kind can scale much faster than a market, and right now we're in a situation where no one is bothering to track pundit predictions or report on pundit track records. Predictions are produced as simple entertainment or as simple political theater, without the slightest fear of accountability.
This site is missing some features, but it looks to me like a starting attempt at what's needed - a Wikipedia-like, user-contributed, low-barrier-to-entry database of all pundit predictions, past and present.
Intrade and the Dow Drop
With today's snapback, the Dow lost 777 and regained 485.
As of this evening, Intrade says the probability of a bailout bill passing by Oct 31st is 85%.
(777-485)/(1-.85) = 1,946. So a bailout bill makes an expected difference of 2000 points on the Dow.
Of course this is a bogus calculation, but it's an interesting one. Not overwhelmingly on-topic for OB, but it involves prediction markets and I didn't see anyone else pointing it out. I hope the bailout fails decisively, so this calculation can be tested.
Buy Now Or Forever Hold Your Peace
The Intrade prediction market is giving Hillary a 53% chance and Obama a 47% chance of winning the Democratic presidential nomination. Hillary is down 7.5 percentage points in just the last day. (Note: Between when I wrote the above, and when I posted this, Hillary went up to 54.)
From what I've read on Intrade, you can fund your account with up to $250 using a credit card, and it should land in your account immediately. (More than this takes time.) Also, remember that you can sell contracts at any time afterward - you don't have to wait months to collect your payout.
If you think that Hillary is going to do better than the polls on Super Tuesday, and you're going to sneer afterward and say that Intrade was "just tracking the polls", buy Hillary now.
If you think that Obama is going to do better than the polls on Super Tuesday, and you're going to gloat about how prediction markets didn't call this surprise in advance, buy Obama now.
If you don't do either, then clearly you do not really believe that you know anything the prediction markets don't. (Or you don't understand expected utility, or your utilities over final outcomes drop off improbably fast in the vicinity of your current wealth minus fifty bucks - you don't have to bet the full $250.) It is free money, going now for anyone who genuinely thinks they know better than the prediction markets what will happen next.
Prediction markets do not have supernatural insight. If they give the candidates fifty-fifty odds, it means that the market collectively doesn't know what will happen next. Even if you're well-calibrated, you get surprised on 90% probabilities one time out of ten.
The point is not that prediction markets are a good predictor but that they are the best predictor. If you think you can do better, why ain'cha rich? Any person, group, or method that does better can pump money out of the prediction markets.
If prediction markets react to polls, they're getting new information, that they didn't predict in advance, which happens. Being the best predictor doesn't make you omniscient.
Everyone's going to find it real easy to make a better prediction afterward, but if you think you can call it in advance, there's FREE MONEY GOING NOW.
Buy now, or forever hold your peace.
The Apocalypse Bet
A problem with betting on engineered superplagues, physics disasters, nanotechnological warfare, or intelligence explosions of both Friendly and unFriendly type, is that all these events are likely to disrupt settlement of trades (to put it mildly). It's not easy to sell a bet that pays off only if the prediction market ceases to exist.
And yet everyone still wants to know the year, month, and day of the Singularity. Even I want to know, I'm just professionally aware that the knowledge is not available.
This morning, I saw that someone had launched yet another poll on "when the Singularity will occur". Just a raw poll, mind you, not a prediction market. I was thinking of how completely and utterly worthless this poll was, and how a prediction market might be slightly less than completely worthless, when it occurred to me how to structure the bet - bet that "settlement of trades will be disrupted / the resources gambled will become worthless, no later than time T".
Suppose you think that gold will become worthless on April 27th, 2020 at between four and four-thirty in the morning. I, on the other hand, think this event will not occur until 2030. We can sign a contract in which I pay you one ounce of gold per year from 2010 to 2020, and then you pay me two ounces of gold per year from 2020 to 2030. If gold becomes worthless when you say, you will have profited; if gold becomes worthlesss when I say, I will have profited. We can have a prediction market on a generic apocalypse, in which participants who believe in an earlier apocalypse are paid by believers in a later apocalypse, until they pass the date of their prediction, at which time the flow reverses with interest. I don't see any way to distinguish between apocalypses, but we can ask the participants why they were willing to bet, and probably receive a decent answer.
I would be quite interested in seeing what such a market had to say. And if the predicted date was hovering around 2080, I would pick up as much of that free money as I dared.
EDIT: Robin Hanson pointed out why this wouldn't work. See comments.
Futuristic Predictions as Consumable Goods
The Wikipedia entry on Friedman Units tracks over 30 different cases between 2003 and 2007 in which someone labeled the "next six months" as the "critical period in Iraq". Apparently one of the worst offenders is journalist Thomas Friedman after whom the unit was named (8 different predictions in 4 years). In similar news, some of my colleagues in Artificial Intelligence (you know who you are) have been predicting the spectacular success of their projects in "3-5 years" for as long as I've known them, that is, since at least 2000.
Why do futurists make the same mistaken predictions over and over? The same reason politicians abandon campaign promises and switch principles as expediency demands. Predictions, like promises, are sold today and consumed today. They produce a few chewy bites of delicious optimism or delicious horror, and then they're gone. If the tastiest prediction is allegedly about a time interval "3-5 years in the future" (for AI projects) or "6 months in the future" (for Iraq), then futurists will produce tasty predictions of that kind. They have no reason to change the formulation any more than Hershey has to change the composition of its chocolate bars. People won't remember the prediction in 6 months or 3-5 years, any more than chocolate sits around in your stomach for a year and keeps you full.
The futurists probably aren't even doing it deliberately; they themselves have long since digested their own predictions. Can you remember what you had for breakfast on April 9th, 2006? I bet you can't, and I bet you also can't remember what you predicted for "one year from now".
View more: Next
= 783df68a0f980790206b9ea87794c5b6)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)