Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Burdensome Details

26 Post author: Eliezer_Yudkowsky 20 September 2007 11:46PM

Followup toConjunction Fallacy

 "Merely corroborative detail, intended to give artistic verisimilitude to an otherwise bald and unconvincing narrative..."
            -- Pooh-Bah, in Gilbert and Sullivan's The Mikado

The conjunction fallacy is when humans rate the probability P(A&B) higher than the probability P(B), even though it is a theorem that P(A&B) <= P(B).  For example, in one experiment in 1981, 68% of the subjects ranked it more likely that "Reagan will provide federal support for unwed mothers and cut federal support to local governments" than that "Reagan will provide federal support for unwed mothers."

A long series of cleverly designed experiments, which weeded out alternative hypotheses and nailed down the standard interpretation, confirmed that conjunction fallacy occurs because we "substitute judgment of representativeness for judgment of probability".  By adding extra details, you can make an outcome seem more characteristic of the process that generates it.  You can make it sound more plausible that Reagan will support unwed mothers, by adding the claim that Reagan will also cut support to local governments.  The implausibility of one claim is compensated by the plausibility of the other; they "average out".

Which is to say:  Adding detail can make a scenario SOUND MORE PLAUSIBLE, even though the event necessarily BECOMES LESS PROBABLE.

If so, then, hypothetically speaking, we might find futurists spinning unconscionably plausible and detailed future histories, or find people swallowing huge packages of unsupported claims bundled with a few strong-sounding assertions at the center.

If you are presented with the conjunction fallacy in a naked, direct comparison, then you may succeed on that particular problem by consciously correcting yourself.  But this is only slapping a band-aid on the problem, not fixing it in general.

In the 1982 experiment where professional forecasters assigned systematically higher probabilities to "Russia invades Poland, followed by suspension of diplomatic relations between USA and USSR" versus "Suspension of diplomatic relations between USA and USSR", each experimental group was only presented with one proposition.  What strategy could these forecasters have followed, as a group, that would have eliminated the conjunction fallacy, when no individual knew directly about the comparison?  When no individual even knew that the experiment was about the conjunction fallacy?  How could they have done better on their probability judgments?

Patching one gotcha as a special case doesn't fix the general problem.  The gotcha is the symptom, not the disease.  That would be as silly as, oh, say, prohibiting box cutters on airplanes.

What could the forecasters have done to avoid the conjunction fallacy, without seeing the direct comparison, or even knowing that anyone was going to test them on the conjunction fallacy?  It seems to me, that they would need to notice the word "and".  They would need to be wary of it - not just wary, but leap back from it.  Even without knowing that researchers were afterward going to test them on the conjunction fallacy particularly.  They would need to notice the conjunction of two entire details, and be shocked by the audacity of anyone asking them to endorse such an insanely complicated prediction.  And they would need to penalize the probability substantially - a factor of four, at least, according to the experimental details.

It might also have helped the forecasters to think about possible reasons why the US and Soviet Union would suspend diplomatic relations.  The scenario is not "The US and Soviet Union suddenly suspend diplomatic relations for no reason", but "The US and Soviet Union suspend diplomatic relations for any reason."

And the subjects who rated "Reagan will provide federal support for unwed mothers and cut federal support to local governments"?  Again, they would need to be shocked by the word "and".  Moreover, they would need to add absurdities - where the absurdity is the log probability, so you can add it - rather than averaging them.  They would need to think, "Reagan might or might not cut support to local governments (1 bit), but it seems very unlikely that he will support unwed mothers (4 bits).  Total absurdity: 5 bits."  Or maybe, "Reagan won't support unwed mothers.  One strike and it's out.  The other proposition just makes it even worse."

Similarly, consider the six-sided die with four green faces and one red face.  The subjects had to bet on the sequence (1) "RGRRR", (2) "GRGRRR", or "GRRRRR" appearing anywhere in 20 rolls of the dice.  65% of the subjects chose "GRGRRR", which is strictly dominated by "RGRRR", since any sequence containing "GRGRRR" also pays off for "RGRRR".  How could the subjects have done better?  By noticing the inclusion?  Perhaps; but that is only a band-aid, it does not fix the fundamental problem.  By explicitly calculating the probabilities?  That would certainly fix the fundamental problem, but you can't always calculate an exact probability.

The subjects lost heuristically by thinking:  "Aha!  Sequence 2 has the highest proportion of green to red!  I should bet on Sequence 2!"  To win heuristically, the subjects would need to think:  "Aha!  Sequence 1 is short!  I should go with Sequence 1!"

They would need to feel a stronger emotional impact from Occam's Razor - feel every added detail as a burden, even a single extra roll of the dice.

Once upon a time, I was speaking to someone who had been mesmerized by an incautious futurist.  (One who adds on lots of details that sound neat.)  I was trying to explain why I was not likewise mesmerized by these amazing, incredible theories.  So I explained about the conjunction fallacy, specifically the "suspending relations +/- invading Poland" experiment.  And he said, "Okay, but what does this have to do with -"  And I said, "It is more probable that universes replicate for any reason, than that they replicate via black holes because advanced civilizations manufacture black holes because universes evolve to make them do it."  And he said, "Oh."

Until then, he had not felt these extra details as extra burdens.  Instead they were corroborative detail, lending verisimilitude to the narrative.  Someone presents you with a package of strange ideas, one of which is that universes replicate.  Then they present support for the assertion that universes replicate.  But this is not support for the package, though it is all told as one story.

You have to disentangle the details.  You have to hold up every one independently, and ask, "How do we know this detail?"  Someone sketches out a picture of humanity's descent into nanotechnological warfare, where China refuses to abide by an international control agreement, followed by an arms race...  Wait a minute - how do you know it will be China?  Is that a crystal ball in your pocket or are you just happy to be a futurist?  Where are all these details coming from?  Where did that specific detail come from?

For it is written:

If you can lighten your burden you must do so.
There is no straw that lacks the power to break your back.

Comments (26)

Sort By: Old
Comment author: joe5 21 September 2007 03:47:40AM 4 points [-]

In some situations, part of the problem may simply be that instead of calculating a joint probability, they are calculating a conditional probability. The effect of mistakenly calculating a conditional probability is that one event may seem highly probable given the other event occurred; I believe this could be a mathematical reason explaining the plausibility effect in some situations.

For example: the probability that USA and USSR suspend diplomatic relations given that Russia invades Poland is probably more likely than the marginal event, USA and USSR suspend diplomatic relations.

Comment author: Felix2 21 September 2007 07:13:55AM -1 points [-]

Ooooo! "Dice roll?" By, God, my good fellow, you mean, "coin flips!"

Comment author: Richard3 21 September 2007 06:50:01PM 7 points [-]

Most of the time, detailed futuristic scenarios are not presented with the intent to say, "exactly this will happen." Rather, they are intended to say, "something like this will happen." Many people have trouble with purely abstract thinking, and benefit from the clarity of specific examples. If you make a general statement about the dangers of reckless scientific experiments (for example), it is likely that many of your listeners either won't be able to connect that to specifics, or will come up with examples in their minds that are very different from what you meant. If you write a novel about it called "Frankenstein," those people will get the point very vividly. The novel has approximately zero chance of coming to pass exactly as written, but it is easier to understand. Unfortunately, the detailed approach carries with it the very real danger that some people will take the fictional details to be the substance of the claim, rather than the abstract principle that they illustrate.

Comment author: Doug_S. 21 September 2007 08:02:34PM 7 points [-]

The variable-width typeface my browser is using typeface makes option 1 look longer than option 2; I had to copy/paste it into Notepad to see that both sequences were equally long. If I hadn't double checked, I would have said sequence 2 is more probable than sequence 1 because it is shorter.

Comment author: Nick_Tarleton 21 September 2007 08:58:56PM 5 points [-]

Echoing Richard, I can see another good reason (and yes, I read your last post) why a more complicated scenario could be assigned a higher probability. Take the USSR example. Suppose that a USSR invasion of Poland is the only reasonably-likely event that could cause a suspension of diplomatic relations. Suppose also that no one would have thought of that as a possibility until it was suggested. Suppose further that once it was suggested as a possibility, the forecasters would realize quickly that it was actually reasonably likely. (I know I've twisted the example beyond all plausibility, but there probably are real situations fitting this form.) Effectively, the forecasters who got the question including Poland have information the others don't - the realization that there is a probable incident that could cause suspension of diplomatic relations - and can rationally assign a higher probability. The forecasters are still at fault for a massive failure of imagination, but not for anything as simple and stupid as the Conjunction Fallacy.

Felix, are you saying that someone shouldn't answer C, because they should consider the context and consider a biased coin? If I knew the coin might be biased, I would answer A, but I don't see what that has to do with any of Eliezer's examples.

Comment author: g 21 September 2007 10:03:26PM 1 point [-]

Nick: Kahneman and Tversky actually mention the mechanism you describe as one cause of the conjunction fallacy, in their 1983 paper that Eliezer linked to. I agree that in the case where the people who see "X" and the people who see "X and Y" are different, this makes it rather unfair to call it a fallacy; K&T don't seem to think so, perhaps because it's clear that people in this situation are either underestimating the probability of X or overestimating that of X&Y or both, so they're making a mistake even if the explanation for that mistake is reasonably non-embarrassing.

I think that Felix is mostly making fun of people who try to think mathematically and who try to answer the question they're asked rather than some different question that they think might make more sense, rather than trying to make a serious point about the nature of biased coins.

Comment author: Felix2 22 September 2007 08:58:50AM -2 points [-]

Nick: Nice spin! :) Context would be important if Eliezer had not asserted as a given that many, many experiments have been done to preclude any influence of context. My extremely limited experience and knowledge of psychological experiments says that there is a 100% chance that such is not a valid assertion. Imagine a QA engineer trying to skate by with the setups of psych experiments you have run in to. But, personal, anecdotal experience aside, it's real easy to believe Eliezer's assertion is true. Most people might have a hard time tuning out context, though, and therefore might have a harder time, both with conjunction fallacy questionnaires and accepting Eliezer's assertion.

g: Yes, keeping in mind that I would be first in line to answer C, myself!

Choice (B) seems a poster boy for "representation". So, that a normal person would choose B is yet another example of this, "probability" question *not* being a question about probability, but about "representation". Which is the point. Why is it hard to imagine that the word, "probable" does not mean, in such questions' contexts, or even, perhaps, in normal human communication, "probable" as a gambler or statistician would think of its meaning? Or, put another way, g, "who try to answer the question they're asked rather..." is an assumptive close. I don't buy it. They were not asked the question you, me, Eliezer, the logician or the autistic thought. They were asked the question that they understood. And, they have the votes to prove it. :)

So far as people making simple logical errors in computing probabilities, as is implied by the word, "fallacy", well, yeah. Your computer can beat you in both logic and probabilities. Just as your calculator can multiply better than you.

Anyway, I believe that the functional equivalent of visual illusions are inherent in anything one might call a mind. I'm just not convinced that this conjunction fallacy is such a case. The experiments mentioned seem more to identify and wonderfully clarify an interesting communications issue - one that probably stands out simply because there are, in these times, many people who make a living answering C.

Comment author: simplicio 10 March 2010 08:21:10PM 3 points [-]

Interesting post!

But: "Similarly, consider the six-sided die with four green faces and one red face."

I seem to be good at catching trivial mistakes.

Comment author: thomblake 10 March 2010 08:34:29PM 0 points [-]

I was going to reply that this is not obviously a mistake, since we might just be ignorant of what the other side is. Then I realized the guesses listed after suggest that the die has four red faces and one green face. Nice catch.

Comment author: JoshSN 30 May 2010 05:31:20PM -1 points [-]

Reagan would be unlikely to provide support to unwed mothers, but maybe as part of a deal in which he got what he wanted, a reduction in expenditures.

Comment author: JDM 04 June 2013 05:42:14PM 3 points [-]

Irrelevant. If there is any possible explanation where he provides the support without that specific deal, it is automatically less likely that both happen, even if the most likely scenario (90%+) of supporting unwed mothers is given said deal. If it is the only possibility, the scenarios would be equally likely; the conjunction could still not possibly be more likely.

Comment author: simplicio 30 May 2010 05:39:18PM 2 points [-]

Similarly, consider the six-sided die with four green faces and one red face. The subjects had to bet on the sequence (1) "RGRRR", (2) "GRGRRR", or "GRRRRR" appearing anywhere in 20 rolls of the dice.

Did you mean to put a (3) in there?

Comment author: TobyBartels 27 July 2010 11:03:06PM *  4 points [-]

add absurdities - where the absurdity is the log probability, so you can add it - rather than averaging them.

This is a very nice measure (which I've seen before) and term for it (which I have not seen).

Eliezer, did you develop this yourself? Should I say to other people ‘Artificial-intelligence researcher Eliezer Yudkowsky defines the absurdity of a proposition to be the opposite of the logarithm of its probability, A = –log P.’ as an introduction before I start to use it? (I threw in a minus sign so that higher probability would be lower absurdity; maybe you were taking the logarithm base 1/2 so you didn't have to do that.)

Comment author: FiftyTwo 04 September 2011 01:37:30PM *  2 points [-]

One possibility is that our intuitive sense of 'is this statement likely to be true' is developed to detect lies by other human beings, rather than to simulate the external world.

For example if someone is trying to convince you of a tribe members bad behaviour, the ability to produce extra details (time/location/etc) makes it more plausible they are truthful rather than lying. However in probability terms each extra detail makes it less likely (e.g. 'probability of bad behaviour' * 'probability of doing it in location x* etc).

Cross-posted in the sequence re-run

Comment author: po8crg 21 September 2012 05:53:25PM 0 points [-]

Depends on people's definition of truth, surely?

If your scoring system for a conjunction statement where one part is true and the other is untrue is to score that as half-true, then the probabilities for the Reagan case are wholly reasonable.

(ie for "Reagan will provide federal support for unwed mothers and cut federal support to local governments", you score 1 for both parts true, 0.5 for one part true and 0 for neither part true, while for "Reagan will provide federal support for unwed mothers" you can only score 1 for true and 0 for false).

If - and it seems reasonable - the intuitive scoring system for a conjunctive statement is similar to this, then the predictions are wholly reasonable.

This means that when there is a real conjunction, we tend to misinterpret it. It seems reasonable then to guess that we don't have an intuitive approach to a true conjunction. If that's the case, then the approach to overcoming the bias is to analyse joint statements to see if a partial truth scores any points - if it does, then our intuition can be trusted more than when it does not.

Comment author: [deleted] 02 January 2013 08:44:34PM 3 points [-]

Logically, if conjunctions made something more likely, disjunctions would make them less likely, which is surprisingly another similar fallacy people succumb to. The more times the word "or" appears in an explanation or theory, the more likely an onlooker will say "Now you're just guessing" and lose confidence in the claim, even though it necessarily becomes more likely.