The conclusion to draw, then, should be a more general one: given an observation O and an explanation E of O, people can over-weight E as a piece of evidence about O's probability.
But is it overweighting to use the fact that the explanation is bad as evidence against the statement being true? A true statement is more likely to have a good explanation than a false one, so it seems that one could do a Bayseian update on the truth of the staement based on the quality of the explanation.
Sounds reasonable. Although I think it's evidence against that kind of updating if it leads one to get a question wrong, one might well get more evidence in favour of that kind of updating in everyday life.
Over the past year, I've noticed a topic where Less Wrong might have a blind spot: public opinion. Since last September I've had (or butted into) five conversations here where someone's written something which made me think, "you wouldn't be saying that if you'd looked up surveys where people were actually asked about this". The following list includes six findings I've brought up in those LW threads. All of the findings come from surveys of public opinion in the United States, though some of the results are so obvious that polls scarcely seem necessary to establish their truth.
If you've read Eliezer's "Hindsight Devalues Science", you're probably starting to feel déjà vu, and might have guessed that I'm bluffing you to make a point. If so, well done — you're quite correct! But before you assume I'm about to repeat Eliezer's trick and stop there, read the other half of my list:
Here's my twist on Eliezer's twist. It is technically true that the list includes six true findings, but the complete list has 12 items, so half of the statements are false. I made up the false statements as fake variations on the true findings, concocted parenthetical rationalizations for them, and randomly mixed the false claims with the true. I expect a lot of people reading this would, after seeing the full list, have a hard time sorting the true from the false without looking at the data — including the people who nodded along in agreement with the first half of the list.
Totally spurious beliefs about public opinion can have a ring of plausibility, especially because it's easy to invent sensible-sounding reasons why they ought to be correct. The availability heuristic presumably plays a role too, with people inferring the state of public opinion from what their friends & acquaintances think, not accounting for how unrepresentative their social network is. In any event, people's opinions of public opinion are often wrong, and it's worth taking a couple of minutes to look for Gallup poll results and the like online before commenting on public opinion.
Sources. On perceptions of whether the benefits of science outweigh its harmful results, see figure 7-11 from chapter 7 of the National Science Foundation's "Science and Engineering Indicators 2012". On party affiliation, see the polls Gallup runs every month. On abortion views, Pew Research Center has statistics for 2007 through 2012. For a breakdown of beliefs about the Vietnam War by age, consult Hazel Erskine's 1970 article "The Polls: Is War a Mistake?" (Public Opinion Quarterly, 34(1), 134–150); Jim Miller extends the data to May 1971, but presents them somewhat differently. On smokers' regrets, see Geoffrey T. Fong et al.'s 2004 article "The near-universal experience of regret among smokers in four countries: Findings from the International Tobacco Control Policy Evaluation Survey" (Nicotine & Tobacco Research, 6(S3), S341–S351). ↩