JoshuaZ comments on 1001 PredictionBook Nights - Less Wrong

51 Post author: gwern 08 October 2011 04:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (49)
Tags: tricycle prediction_markets predictions calibration rationality standard_biases

Comments (49)

You are viewing a single comment's thread. Show more comments above.

Comment author: torekp 10 October 2011 10:44:28PM 3 points [-]

I am considerably more skeptical of op-eds and other punditry, after tracking the rare clear predictions they made

In the case of a few well-studied pundits you should examine the evidence gathered by other prediction trackers. Some pundits are well outside the dumb luck range on a ten-point scale:

The best? Paul Krugman with a PVS of 8.2 (You can see a screenshot of his score sheet to the right. Note: Score sheets for each of the pundits are in the full text document).

The worst? Cal Thomas, with a PVS of -8.7 (You read that right. Negative eight point seven...).

Kinda surprising to me that you can beat dumb luck in inaccuracy. I hope they do a followup.

Comment author: JoshuaZ 10 October 2011 11:01:05PM 0 points [-]

Kinda surprising to me that you can beat dumb luck in inaccuracy.

It shouldn't be. Assume that your pundits in general do no better than chance. In a large sample, some of them are going to have to have to do really badly. Even if your pool on average is better than chance one should still expect a few much worse.

That said, even given that, -8.7 by their metric looks really badly.

According to that study, being a lawyer by training was one of the things that caused predictors to do badly. Note that Cal Thomas doesn't fall into that category.

Powered by Reddit Powered by Reddit