Kaj_Sotala comments on I Was Not Almost Wrong But I Was Almost Right: Close-Call Counterfactuals and Bias - Less Wrong

54 Post author: Kaj_Sotala 08 March 2012 05:39AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (39)

You are viewing a single comment's thread. Show more comments above.

Comment author: Kaj_Sotala 11 March 2012 11:11:42AM *  2 points [-]

I quoted this excerpt from Tetlock (1998) in the post, did you not find it helpful?

Theoretically, high need-for-closure individuals are characterized by two tendencies: urgency which inclines them to 'seize' quickly on readily available explanations and to dismiss alternatives and permanence which inclines them to 'freeze' on these explanations and persist with them even in the face of formidable counterevidence. In the current context, high need-for-closure individuals were hypothesized to prefer simple explanations that portray the past as inevitable, to defend these explanations tenaciously when confronted by dissonant close-call counterfactuals that imply events could have unfolded otherwise, to express confidence in conditional forecasts that extend these explanations into the future, and to defend disconfirmed forecasts from refutation by invoking second-order counterfactuals that imply that the predicted events almost happened.

The papers I referenced (see the end of the post for links) briefly discuss how this was measured. For instance, Tetlock 1998:

The Need for Closure Scale was adapted from a longer scale developed by Kruglanski and Webster (1996) and included the following eight items: "I think that having clear rules and order at work is essential for success"; "Even after I have made up my mind about something, I am always eager to consider a different opinion' l; " I dislike questions that can be answered in many different ways''; "I usually make important decisions quickly and confidently"; "When considering most conflict situations, I can usually see how both sides could be right"; "It is annoying to listen to someone who cannot seem to make up his or her mind"; "I prefer interacting with people whose opinions are very different from my own"; and "When trying to solve a problem I often see so many possible options that it is confusing."1 Experts rated their agreement with each item on 9-point disagree-agree scales.

Incidentally, Tetlock 1998 also used another measure that's theoretically different from need-for-closure, namely integrative complexity.

Integrative complexity should be negatively correlated with need for closure. It implies not only a willingness to entertain contradictory ideas but also an interest in generating, testing, and revising integrative cognitions that specify flexible boundary conditions for contradictory hypotheses. The two constructs— need for closure and integrative complexity—are, however, measured in very different ways: a traditional selfreport personality scale in the case of need for closure and an open-ended thought-sampling procedure requiring content analysis in the case of integrative complexity. Given the severe problems of method variance that have bedeviled cognitivestyle research over the past 50 years (Streufert, 1997), a major advantage of the present study is the inclusion of methodologically dissimilar but conceptually overlapping procedures for assessing cognitive style. [...]

The integrative complexity measure was derived from open-ended responses to a request to reflect on 20th-century history. The following question was used: "Did the 20th century have to be as violent as it has been?" We assured respondents that we understood that many books had been written on this subject and that many more undoubtedly would be written. Our goal was just to get a quick sense for the factors that they deemed most decisive in shaping the general course of events (the sort of shorthand answer they might give a respected colleague in a different discipline at a social occasion). Integrative complexity was coded on a 7-point scale in which scores of 1 were given to statements that identified only causal forces that increased or decreased the likelihood of the specified outcomes (e.g., "Nationalism and mass production of weapons guaranteed disaster"), scores of 3 were assigned to statements that identified causal forces with contradictory effects (e.g., ' "Iwentieth-century history will be remembered not only for the destructive forces unleashed—totalitarianism and weapons of mass destruction— but also for the initial steps toward global governance"), scores of 5 were assigned to statements that tried to integrate two contradictory causal forces (e.g., "Wars can be caused by being too tough or too soft and it is really hard to strike the right balance—that's the big lesson of 20th-century diplomacy"), and scores of 7 placed the problem of integrating causal forces into a broader systemic frame of reference (e.g., "YJU could argue that we got off lucky and escaped nuclear war or that we were incredibly unlucky and wound up with a holocaust that was the product of one man's obsession. How you look at it is a matter of personal temperament and philosophy. My guess is that we are running about par for the course"). Intercoder agreement was .85 between two raters who were blind to both the hypotheses being tested and to the sources of the material.

Tetlock 1998 found the results between the two measures of need-for-closure and integrative complexity to be highly similar, in that individuals with a high need-for-closure scored low on integrative complexity, and vice versa. He combined the results of the two measures to a single variable in analyzing the results of that study. IIRC, in later studies only need-for-closure was used.