Here's a video of a presentation by Hal Arkes, one of the top world experts in debiasing, Emeritus Professor at Ohio State, and Intentional Insights Advisory Board member, on dealing with hindsight bias and overconfidence. This was at a presentation hosted by Intentional Insights and the Columbus, OH Less Wrong group. It received high marks from local Less Wrongers, so I thought I'd share it here.

Back in August I ran a Caplan Test (or more commonly an "Ideological Turing Test") both on Less Wrong and in my local rationality meetup. The topic was diet, specifically: Vegetarian or Omnivore?

If you're not familiar with Caplan Tests, I suggest reading Palladias' post on the subject or reading Wikipedia. The test I ran was pretty standard; thirteen blurbs were presented to the judges, selected by the toss of a coin to either be from a vegetarian or from an omnivore, and also randomly selected to be genuine or an impostor trying to pass themselves off as the alternative. My main contribution, which I haven't seen in previous tests, was using credence/probability instead of a simple "I think they're X".

I originally chose vegetarianism because I felt like it's an issue which splits our community (and particularly my local community) pretty well. A third of test participants were vegetarians, and according to the 2014 census, only 56% of LWers identify as omnivores.

Before you see the results of the test, please take a moment to say aloud how well you think you can do at predicting whether someone participating in the test was genuine or a fake.

.

.

.

.

.

.

.

.

.

.

.

.

.

If you think you can do better than chance you're probably fooling yourself. If you think you can do significantly better than chance you're almost certainly wrong. Here are some statistics to back that claim up.

I got 53 people to judge the test. 43 were from LessWrong, and 10 were from my local group. Averaging across the entire group, 51.1% of judgments were correct. If my Chi^2 math is correct, the p-value for the null hypothesis is 57% on this data. (Note that this includes people who judged an entry as 50%. If we don't include those folks the success rate drops to 49.4%.)

In retrospect, this seemed rather obvious to me. Vegetarians aren't significantly different from omnivores. Unlike a religion or a political party there aren't many cultural centerpieces to diet. Vegetarian judges did no better than omnivore judges, even when judging vegetarian entries. In other words, in this instance the minority doesn't possess any special powers for detecting other members of the in-group. This test shows null results; the thing that distinguishes vegetarians from omnivores is not familiarity with the other sides' arguments or culture, at least not to the degree that we can distinguish at a glance.

More interesting, in my opinion, than the null results were the results I got on the calibration of the judges. Back when I asked you to say aloud how good you'd be, what did you say? Did the last three paragraphs seem obvious? Would it surprise you to learn that not a single one of the 53 judges held their guesses to a confidence band of 40%-60%? In other words, every single judge thought themselves decently able to discern genuine writing from fakery. The numbers suggest that every single judge was wrong.

(The flip-side to this is, of course, that every entrant to the test won! Congratulations rationalists: signs point to you being able to pass as vegetarians/omnivores when you try, even if you're not in that category. The average credibility of an impostor entry was 59%, while the average credibility of a genuine response was 55%. No impostors got an average credibility below 49%.)

Using the logarithmic scoring rule for the calibration game we can measure the error of the community. The average judge got a score of -543. For comparison, a judge that answered 50% ("I don't know") to all questions would've gotten a score of 0. Only eight judges got a positive score, and only one had a score higher than 100 (consistent with random chance). This is actually one area where Less Wrong should feel good. We're not at all calibrated... but for this test at least, the judges from the website were much better calibrated than my local community (who mostly just lurk). If we separate the two groups we see that the average score for my community was -949, while LW had an average of -448. Given that I restricted the choices to multiples of 10, a random selection of credences gives an average score of -921.

In short, the LW community didn't prove to be any better at discerning fact from fiction, but it was significantly less overconfident. More de-biasing needs to be done, however! The next time you think of a probability to reflect your credence, ask yourself "Is this the sort of thing that anyone would know? Is this the sort of thing I would know?" That answer will probably be "no" a lot more than it feels like from the inside.

Full data (minus contact info) can be found here.

Those of you who submitted a piece of writing that I used, or who judged the test and left their contact information: I will be sending out personal scores very soon (probably by this weekend). Deep apologies regarding the delay on this post. I had a vacation in late August and it threw off my attention to this project.

EDIT: Here's a histogram of the identification accuracy. 

EDIT 2: For reference, here are the entries that were judged.

EDIT: Image now visisble!

From Anders Sandberg:

Another piece examining predictive performance, this time in the pharmaceutical industry. How well can industry experts predict sales?

You guessed it, not very well. Not even when data really accumulated.

Large pharma has less bias than small companies, but the variance still overshadows everything.

First, most consensus forecasts were wrong, often substantially. And although consensus forecasts improved over time as more information became available, accuracy remained an issue even several years post-launch. More than 60% of the consensus forecasts in our data set were either over or under by more than 40% of the actual peak revenues (Fig. a). Although the overall median of the data set was within 4%, the distribution is wide for both under- and overestimated forecasts. Furthermore, a significant number of consensus forecasts were overly optimistic by more than 160% of the actual peak revenues of the product.

The unanswered question in this analysis is what companies and investors ought to be doing to forecast better. We do not offer a complete answer here, but we have thoughts based on our analysis.

Beware the wisdom of the crowd. The 'consensus' consists of well-compensated, focused professionals who have many years of experience, and we have shown that the consensus is often wrong. There should be no comfort in having one's own forecast being close to the consensus, particularly when millions or billions of dollars are on the line in an investment decision or acquisition situation.

Broaden the aperture on what the future could look like, and rapidly adapt to new information. Much of the divergence between a forecast and what actually happens is due to the emergence of a scenario that no one foresaw: a new competitor, unfavourable clinical data or a more restrictive regulatory environment. Companies need to fight their own inertia and the tendency to make only incremental shifts in forecasting and resourcing.

Try to improve. It appears that some companies and analysts may be better at forecasting than others (see Supplementary information S1 (box)). We suspect there is no magic bullet to improving the accuracy of forecasts, but the first step is conducting a self-assessment and recognizing that there may be a capability issue that needs to be addressed.

I've just been through the proposal for the Dartmouth AI conference of 1956, and it's a surprising read. All I really knew about it was its absurd optimism, as typified by the quote:

An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

But then I read the rest of the document, and was... impressed. Go ahead and read it, and give me your thoughts. Given what was known in 1955, they were grappling with the right issues, and seemed to be making progress in the right directions and have plans and models for how to progress further. Seeing the phenomenally smart people who were behind this (McCarthy, Minsky, Rochester, Shannon), and given the impressive progress that computers had been making in what seemed very hard areas of cognition (remember that this was before we discovered Moravec's paradox)... I have to say that had I read this back in 1955, I think the rational belief would have been "AI is probably imminent". Some overconfidence, no doubt, but no good reason to expect these prominent thinkers to be so spectacularly wrong on something they were experts in.

One of the main Eliezer Sequences, consisting of dozens of posts, is How To Actually Change Your Mind. Looking at all those posts, one gets the feeling that changing one’s mind must be Really Hard. But maybe it doesn't have to be that hard. I think it would much easier to change your mind, if you instinctively thought that your best ideas are almost certainly still far from the truth. Most of us are probably aware of the overconfidence bias, but there hasn't been much discussion on how to practically reduce overconfidence in our own ideas.

I offer two suggestions in that vein for your consideration.

1. Take the outside view. Recall famous scientists and philosophers of the past, and how far off from the truth their ideas were, and yet how confident they were in their ideas. Realize that they are famous because, in retrospect, they were more right than everyone else of their time, and there are countless books filled with even worse ideas. How likely is it that your ideas are the best of our time? How likely is it that the best ideas of our time are fully correct (as opposed to just a bit closer to the truth)?

2. Take a few days to learn some cryptology and then design your own cipher. Use whatever tricks you can find and make it as complicated as you want. Feel your confidence in how unbreakable it must be (at least before the Singularity occurs), and then watch it taken apart by an expert in minutes. Now feel the sense of betrayal against your “self-confidence module” and vow “never again”.

http://www.ted.com/talks/kathryn_schulz_on_being_wrong.html

Kathryn Schulz is a self-identified "Wrongologist" (in fact, @wrongologist is her user name on Twitter).  She has written a popular book ("Being Wrong: Adventures in the Margin of Error", web site) and also writes the Slate column 'The Wrong Stuff'.  Her TED talk covers the problem of disagreement, the nature of belief, overconfidence bias and how to actually change your mind.  She maintains that most folks actively avoid the unpleasant feeling of "being wrong", which is an important point I have not seen before (but see The Importance of Saying 'Oops' and Crisis of Faith).  Unfortunately, she does not discuss reasoning about uncertainty, so her arguments against 'the feeling of right' end up seeming rather shallow.

Discuss her TED talk here. (Her broader work is also obviously on topic.)