1001 PredictionBook Nights

gwern

I explain what I've learned from creating and judging thousands of predictions on personal and real-world matters: the challenges of maintenance, the limitations of prediction markets, the interesting applications to my other essays, skepticism about pundits and unreflective persons' opinions, my own biases like optimism & planning fallacy, 3 very useful heuristics/approaches, and the costs of these activities in general.

Plus an extremely geeky parody of Fate/Stay Night.

This essay exists as a large section of my page on predictions markets on gwern.net: http://www.gwern.net/Prediction%20markets#1001-predictionbook-nights

I explain what I've learned from creating and judging thousands of predictions on personal and real-world matters: the challenges of maintenance, the limitations of prediction markets, the interesting applications to my other essays, skepticism about pundits and unreflective persons' opinions, my own biases like optimism & planning fallacy, 3 very useful heuristics/approaches, and the costs of these activities in general.

Plus an extremely geeky parody of Fate/Stay Night.

This essay exists as a large section of my page on predictions markets on gwern.net: http://www.gwern.net/Prediction%20markets#1001-predictionbook-nights

I am considerably more skeptical of op-eds and other punditry, after tracking the rare clear predictions they made

In the case of a few well-studied pundits you should examine the evidence gathered by other prediction trackers. Some pundits are well outside the dumb luck range on a ten-point scale:

The best? Paul Krugman with a PVS of 8.2 (You can see a screenshot of his score sheet to the right. Note: Score sheets for each of the pundits are in the full text document).

The worst? Cal Thomas, with a PVS of -8.7 (You read that right. Negative eight point seven...).

Kinda surprising to me that you can beat dumb luck in inaccuracy. I hope they do a followup.

Yes, I remember that study - it wasn't as long term as I would like, and I always wonder about the quality of a study conducted by students, but it was interesting anyway.

7Eugine_Nier15y

Since the study focused on the period around the 2008 elections, which the Democrats won on nearly all levels, and since most pundits tend to be biased towards believing that what they wish would happen will happen, it's not surprising that liberals' predictions did better and some conservatives scored worse than random. I suspect we'd see the trend go the other way for say predictions about the 2010 midterms. The fundamental problem is that the predictions weren't independent.

0JoshuaZ15y

It shouldn't be. Assume that your pundits in general do no better than chance. In a large sample, some of them are going to have to have to do really badly. Even if your pool on average is better than chance one should still expect a few much worse. That said, even given that, -8.7 by their metric looks really badly. According to that study, being a lawyer by training was one of the things that caused predictors to do badly. Note that Cal Thomas doesn't fall into that category.

72

1001 PredictionBook Nights

72

72

72

1001 PredictionBook Nights

72

72