Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Eliezer_Yudkowsky comments on Scope Insensitivity - Less Wrong

46 Post author: Eliezer_Yudkowsky 14 May 2007 02:53AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (44)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Eliezer_Yudkowsky 15 May 2007 06:13:06PM 5 points [-]

I perceive that I've neglected to convey the existence of a gigantic body of supporting evidence.

Michael Sullivan, see e.g. http://www.sas.upenn.edu/~baron/cv1.htm:

Embedding. Kahneman and Knetsch (1992) asked some subjects their WTP for improved disaster preparedness and other subjects their WTP for improved rescue equipment and personnel. The improved equipment and personnel were thus ``embedded in'' the improved disaster preparedness, so the preparedness included the equipment and personnel, and other things too. WTP was, however, about the same for the larger good and for the smaller good included in it. Kahneman and Knetsch called this the ``perfect embedding effect,'' presumably because a demonstration of it requires perfect equality of WTP of the two goods. When subjects were asked their WTP for the smaller good after they had just been asked about the larger one, they gave much smaller values for the smaller good than for the larger one, and much smaller values than those given by subjects who were asked just about the smaller good. This order effect is called the ``regular embedding effect.'' It demonstrates that a good seen as embedded in a larger good has reduced value. Kemp & Maxwell (1993) replicated this regular embedding effect, starting with a broad spectrum of public goods, and narrowing the good down in several steps, obtaining WTPs for an embedded good that were 1/300 of WTP for the same good in isolation.

Adding up. In a related demonstration, Diamond et al. (1993), asked subjects their WTP values for preventing timber harvesting in federally protected wilderness areas. WTP for the prohibition in three areas was not much (if any) higher than WTP for prohibition in one of the areas alone. This result cannot be explained by assuming that subjects thought that protection of one area was sufficient: when they were asked their WTP to protect one area assuming that another was already protected, or a third area assuming that two were protected, their WTP values were just as high as those for protecting the first area. More generally, in this kind of ``adding-up effect,'' respondents are asked their WTP for good A (e.g., a single wilderness area), for good B assuming that good A has been provided already, and for goods A and B together. WTP for A and B together is much lower than the sum of the WTP for A and for B (with A provided). Schulze, McClelland, & Lazo (1994) found similar results in a within-subject design: each subject rated A, B, and A and B together.

Comment author: Will_Newsome 18 November 2011 02:28:37PM *  8 points [-]

I perceive that I've neglected to convey the existence of a gigantic body of supporting evidence.

There is much counterevidence in the literature as well, but more importantly the literature does not clearly suggest the extent to which people are scope sensitive when they are (which is often), nor does it suggest what normative sensitivity might look like given the complexities of the decision problems and of human preferences. The literature doesn't tell us the extent to which self-identifying total-utilitarian-style altruists in particular are scope sensitive, nor what methods of assigning WTP values they use. Whether or not their decisions are normative according to their professed optimization criteria, and more importantly whether their decisions are more or less normative than a naive "shut up and multiply the salient numbers" approach, is unknown.

A naive total utilitarian approach is clearly lacking. There are always hidden and unmentioned complexities like predetermined ecological niche sizes, i.e. 50 saved birds will quickly breed so as to fill a niche whereas 5,000 birds will remain at the limits. The difference between 1,000 out of 50,000 versus 1,000 out of 2,000 human lives saved is a substantial difference: realistic attempts at either will look very different from each other. Logarithmic scaling is common and can be a natural result of (implicit) consideration of conjunctions, exaggerations, credibility calculations (like whether it'd be easy or difficult to fake a positive result), baselines, opportunity costs, and so on; it is unclear what a normative evaluation of disutility from wars of various casualties would look like, but logarithmicness doesn't seem obviously wrong. (The different framings in the original paper suggest different metrics for evaluation; there's no reason to expect consistent valuations across levels of organization. "Deaths per day" offers an uncomplicated metric, "magnitude of war" prompts highly complex evaluations where log-normal distributions are significant.) Lives (alleged) to be saved affect utility calculations only additively, less than do estimated probabilities of internal successes or failures. In brief, a substantial amount of information is not represented by the numbers, and so substantial deviations from naive additive WTP values should be expected.

Naive total utilitarianism is a fast and frugal algorithm which ignores many considerations and makes no attempt to reach normative decisions. Whether it's more or less consistent with total utilitarians' values than more intuitive approaches is unclear, and which to prefer in the absence of such information is likewise unclear. Finally, don't forget that meta-level uncertainty about total utilitarianism should be taken into account.

ETA: I should highlight that there is much variance between subjects and between studies. I do not argue that some subjects in some studies don't simply purchase moral satisfaction or the like (though the research indicates this is uncommon), but I do argue that some non-negligible number of subjects in some non-negligible number of studies might be more effective altruists than any explicitly algorithm/equation-centered approach would allow for.

ETA2: The above analysis assumes that people's responses to surveys about why/how they made a decision or what affected them isn't generally correlated much with their actual decision processes. This assumption is reasonable and isn't necessary but it's not overwhelmingly disjunctive.