gwern comments on 1001 PredictionBook Nights - Less Wrong

51 Post author: gwern 08 October 2011 04:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (49)
Tags: tricycle prediction_markets predictions calibration rationality standard_biases

Comments (49)

You are viewing a single comment's thread. Show more comments above.

Comment author: torekp 10 October 2011 10:44:28PM 3 points [-]

I am considerably more skeptical of op-eds and other punditry, after tracking the rare clear predictions they made

In the case of a few well-studied pundits you should examine the evidence gathered by other prediction trackers. Some pundits are well outside the dumb luck range on a ten-point scale:

The best? Paul Krugman with a PVS of 8.2 (You can see a screenshot of his score sheet to the right. Note: Score sheets for each of the pundits are in the full text document).

The worst? Cal Thomas, with a PVS of -8.7 (You read that right. Negative eight point seven...).

Kinda surprising to me that you can beat dumb luck in inaccuracy. I hope they do a followup.

Comment author: gwern 11 October 2011 08:15:35PM 2 points [-]

Yes, I remember that study - it wasn't as long term as I would like, and I always wonder about the quality of a study conducted by students, but it was interesting anyway.

Comment author: [deleted] 11 October 2011 08:19:31PM *  3 points [-]

The last time I cited this study, I remember that their sample size was well under thirty for each of their pundits. At that level, what's the point of statistics?

Comment author: gwern 25 August 2012 11:29:13PM *  2 points [-]

If the effect size is large enough, 30 observations is plenty & enough to do stats on. Go through a power calculation sometime with, say, d=0.7.

Powered by Reddit Powered by Reddit