Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Unnamed comments on So You Think You're a Bayesian? The Natural Mode of Probabilistic Reasoning - Less Wrong

48 Post author: Matt_Simpson 14 July 2010 04:51PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (79)

You are viewing a single comment's thread.

Comment author: Unnamed 14 July 2010 06:58:29PM *  24 points [-]

This frequencies vs. probabilities issue is one of the controversies in heuristics & biases research, and Kahneman, Tversky, and others dispute Gigerenzer's take. For instance, here (pdf, see p. 8) is what Gilovich & Griffin have to say in their introduction to the book Heuristics and Biases (emphasis added):

In fact, presenting frequencies rather than probabilities sometimes makes judgment distinctly worse (e.g., Griffin & Buehler, 1999; Treadwell & Nelson, 1996), sometimes makes judgments distinctly better (e.g., Tversky & Kahneman, 1983; Koehler, Brenner, & Tversky, 1997) and quite often leaves the quality of judgment largely unchanged (Brenner, Koehler, Liberman, & Tversky, 1996; Griffin & Buehler, 1999). Even more troublesome for the evolution/frequency argument, Kahneman and Tversky’s original explanation of the probability—frequency discrepancy (Kahneman & Tversky, 1982a; Tversky & Kahneman, 1983) provides a unified account of when frequency formats improve judgments and when they do not (e.g., Sloman, Slovak, & Over, 2000).

Critics claim that assessments of single-event probabilities are unnatural, and that only a frequency format is consistent with how the mind works (Cosmides & Tooby, 1996; Gigerenzer, 1991b, 1994; Pinker, 1997). Kahneman and Tversky argued, in contrast, that representing problems in terms of frequencies tends to evoke mental models that facilitate the detection of set inclusion relations and thus improves judgment — and this view has received considerable support from the studies of Sloman and others (e.g., Evans, Handley, Perham, Over, & Thompson, 2000; Girotto & Gonzalez, 2001; Sloman & Over, in press; Sloman et al., 2000).

Comment author: [deleted] 15 July 2010 02:21:24PM 6 points [-]

Indeed, there seems to have been something of a feud between Kahneman and Gigerenzer. See Kahneman's response to Gigerenzer, and Gigerenzer's counter-response.

Comment author: Matt_Simpson 18 July 2010 06:50:49AM *  19 points [-]

Thanks for the links!

Reading both of these papers sent me on a trawl through the literature. Kahneman's paper sparked it. He reported an experiment to test Gigerenzer's hypothesis that recasting the problem in terms of frequencies reduces or eliminates the prevalence of the conjunction fallacy (pg 586-7). The results, according to Kahneman, confirm his hypothesis that frequencies just cue people to think in terms of set relations and reject Gigerenzer's hypothesis that people are natural frequentists.

In an unpublished manuscript (that I did not read), Hertwick (1997) challenged Kahneman's interpretation of his experiment, arguing that the language Kahneman used led subjects to misinterpret "and" to be disjunctive - i.e., when Kahneman asked how many out of 1000 women fitting the Linda description are "feminists and active bank tellers," the subjects interpreted this as "how many are feminists and how many bank tellers." Hertwick ran an experiment to test this, confirming his hypothesis.

Then Hertwick, Kahneman, and Mellers wrote an "adversarial collaboration" where Mellers arbitrated the disagreement between Hertwick and Kahneman (pdf). They ran 3 experiments to test the different interpretations that Hertwick and Kahneman were giving the data. The experiments didn't resolve the disagreement completely, but both parties moved closer to the other's position.

Finally, Tentori, Bonini, and Osherson devised a way to directly test whether subjects were misinterpreting the term "and" while they gave their subjects Linda-like problems (pdf). They ran two treatments - one with the probability language and the other with the frequency language. In both treatments, the majority of subjects interpreted "and" conjunctively, and those who interpreted it conjunctively still fell prey to the conjunction fallacy. There was no difference between treatments on the interpretation, and of those who interpreted "and" conjunctively, there was a minor but statistically insignificant difference in the prevalence of conjunction errors between the frequency and probability treatments.

Mea culpa. We aren't natural frequentists. When the frequency language helps, it seems to be due to cuing the set inclusion relation, as Kahneman argued. I don't think the issue is completely settled - in particular I think the last paper doesn't quite get at the crux of the issue - but the evidence is pointing in the direction of Kahneman's hypothesis.

Comment author: XiXiDu 24 July 2010 06:56:12PM 2 points [-]
Comment author: Unnamed 16 July 2010 06:17:37AM 0 points [-]

Good links. Kahneman's response is worth reading, especially if you find yourself somewhat persuaded by Gigerenzer's point of view.

Comment author: Matt_Simpson 14 July 2010 11:36:31PM 3 points [-]

Thanks for the link. I've edited the post to let people know about the controversy.

Comment author: arfle 14 July 2010 08:17:59PM 1 point [-]

The pdf here is well worth reading! Thanks.