hard-to-fake evidence
The CIA report, which I haven't read, is pretty hard to fake. But it may not be so useful if we don't know how good is the baseline of CIA analysts and the questions. Tetlock says that it's very easy to write very simple computer programs that beat all of the people he assessed.
I read somewhere that both the CIA analysts and Bueno de Mesquita got 90%, whatever that means, but that he was more precise. I don't know how well-calibrated 90% is, but I suspect that the CIA analysts had more humility than any of Tetlock's experts, perhaps because they considered themselves cogs, rather than celebrities.
I assume he really did publish lots of predictions. But I wouldn't be shocked if he's blatantly lying and they're just wrong. Here's an example of someone contradicted about his predictions by his publications. In 1987, Michael Lewis begins his epilogue to Liar's Poker with "I didn't think the firm was doomed. I didn't think that Wall Street would collapse." Today, he writes "In the two decades since then, I had been waiting for the end of Wall Street." These quotes are not entirely fair, being cherry-picked from the midst of qualifications and explanations. But the similar language is striking.
(When I invoke Tetlock, I'm talking about his book. I don't know what he wrote about BdM.)
Tetlock says that it's very easy to write very simple computer programs that beat all of the people he assessed.
This is probably related to the fallacy of college admissions. Everyone in admissions thinks they can do better a job of predicting college success by using more criteria and their own professional judgement, and in every test using the SAT alone does better in aggregate.
I stumbled upon an article called The New Nostradamus, reporting of a game-theoretic model which predicts political outcomes with startling effectiveness. The results are very impressive. However, the site hosting the article is unfamiliar to me, so I'm not certain of the article's verity, but a quick Google seems to support the claims, at least on a superficial skimming. Here's his TED talk. The model seems almost too good to be true, though. Anybody know more?
Some choice bits from the article:
The claim:
The results:
Gets good money for it:
The method should be of special interest to the OB/LW audience, as it brings to mind discussions about self-deception and evolutionary vs. acknowledged goals and behavior: