Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Eliezer_Yudkowsky comments on Torture vs. Dust Specks - Less Wrong

35 Post author: Eliezer_Yudkowsky 30 October 2007 02:50AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (482)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Eliezer_Yudkowsky 30 October 2007 11:28:00PM 6 points [-]

I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.

Some comments:

While some people tried to appeal to non-linear aggregation, you would have to appeal to a non-linear aggregation which was non-linear enough to reduce 3^^^3 to a small constant. In other words it has to be effectively flat. And I doubt they would have said anything different if I'd said 3^^^^3.

If anything is aggregating nonlinearly it should be the 50 years of torture, to which one person has the opportunity to acclimate; there is no individual acclimatization to the dust specks because each dust speck occurs to a different person. The only person who could be "acclimating" to 3^^^3 is you, a bystander who is insensitive to the inconceivably vast scope.

Scope insensitivity - extremely sublinear aggregation by individuals considering bad events happening to many people - can lead to mass defection in a multiplayer prisoner's dilemma even by altruists who would normally cooperate. Suppose I can go skydiving today but this causes the world to get warmer by 0.000001 degree Celsius. This poses very little annoyance to any individual, and my utility function aggregates sublinearly over individuals, so I conclude that it's best to go skydiving. Then a billion people go skydiving and we all catch on fire. Which exact person in the chain should first refuse?

I may be influenced by having previously dealt with existential risks and people's tendency to ignore them.

Comment author: MrHen 11 February 2010 01:51:08AM 1 point [-]

I will admit, that was a pretty awesome lesson to learn. Marcello's reasoning had it click in my head but the kicker that drove the point home was scaling it to 3^^^^3 instead of 3^^^3.

Comment author: Aharon 17 December 2010 09:01:37PM 3 points [-]

I think I understand why one should derive the conclusion to torture one person, given these premises.

What I don't understand is the premises. In the article about scope insensitivity you linked to, it was very clear that the scope of things made things worse. I don't understand why it should be wrong to round down the dust speck, or similar very small disutilities, to zero - Basically, what Scott Clark said: 3^^^3*0 disutilities = 0 disutility.

Comment author: dlthomas 11 October 2011 10:16:03PM 7 points [-]

Rounding to zero is odd. In the absence of other considerations, you have no preference whether or not people get a dust speck in their eye?

It is also in violation of the structure of the thought experiment - a dust speck was chosen as the least bad bad thing that can happen to someone. If you would round it to zero, then you need to choose slightly worse thing - I can't imagine your intuitions will be any less shocked by preferring torture to that slightly worse thing.

Comment author: lessdazed 11 October 2011 10:56:56PM 2 points [-]

a dust speck was chosen as the least bad bad thing that can happen to someone.

That was a mistake, since so many people round it to zero.

Comment author: dlthomas 11 October 2011 11:14:46PM 2 points [-]

It seems to have been. Since the criteria for the choice was laid out explicitly, though, I would have hoped that more people would notice that the thought experiment they solved so easily was not actually the one they had been given, and perform the necessary adjustment. This is obviously too optimistic - but perhaps can serve itself as some kind of lesson about reasoning.

Comment author: Aharon 29 October 2011 01:14:24PM 0 points [-]

I conceed that it is reasonable within the constraints of the thought experiment. However, I think it should be noted that this will never be more than a thought experiment and that if real world numbers and real world problems are used, it becomes less clear cut, and the intuition of going against the 50 years torture is a good starting point in some cases.

Comment author: dlthomas 11 October 2011 10:12:31PM *  5 points [-]

While some people tried to appeal to non-linear aggregation, you would have to appeal to a non-linear aggregation which was non-linear enough to reduce 3^^^3 to a small constant.

Sum(1/n^2, 1, 3^^^3) < Sum(1/n^2, 1, inf) = (pi^2)/6

So an algorithm like, "order utilities from least to greatest, then sum with a weight if 1/n^2, where n is their position in the list" could pick dust specks over torture while recommending most people not go sky diving (as their benefit is outweighed by the detriment to those less fortunate).

This would mean that scope insensitivity, beyond a certain point, is a feature of our morality rather than a bias; I am not sure my opinion of this outcome.

That said, while giving an answer to the one problem that some seem more comfortable with, and to the second that everyone agrees on, I expect there are clear failure modes I haven't thought of.

Edited to add:

This of course holds for weights of 1/n^a for any a>1; the most convincing defeat of this proposition would be showing that weights of 1/n (or 1/(n log(n))) drop off quickly enough to lead to bad behavior.

Comment author: dlthomas 11 November 2011 10:48:49PM *  0 points [-]

On recently encountering the wikipedia page on Utility Monsters and thence to the Mere Addition Paradox, it occurs to me that this seems to neatly defang both.

Edited - rather, completely defangs the Mere Addition Paradox, may or may not completely defang Utility Monsters depending on details but at least reduces their impact.

Comment author: private_messaging 06 August 2013 08:00:23PM *  2 points [-]

While some people tried to appeal to non-linear aggregation, you would have to appeal to a non-linear aggregation which was non-linear enough to reduce 3^^^3 to a small constant. In other words it has to be effectively flat. And I doubt they would have said anything different if I'd said 3^^^^3.

And why should they consider 3^^^^3 differently, if their function asymptotically approaches a limit? Besides, human utility function would take the whole, and then perhaps consider duplicates, uniqueness (you don't want your prehistoric tribe to lose the last man who knows how to make a stone axe), and so on, rather than evaluate one by one and then sum.

Scope insensitivity - extremely sublinear aggregation by individuals considering bad events happening to many people - can lead to mass defection in a multiplayer prisoner's dilemma even by altruists who would normally cooperate. Suppose I can go skydiving today but this causes the world to get warmer by 0.000001 degree Celsius. This poses very little annoyance to any individual, and my utility function aggregates sublinearly over individuals, so I conclude that it's best to go skydiving. Then a billion people go skydiving and we all catch on fire. Which exact person in the chain should first refuse?

The false allure of oversimplified morality is in ease of inventing hypothetical examples where it works great.

One could, of course, posit a colder planet. Most of the population would prefer that planet to be warmer, but if the temperature rise exceeds 5 Celsius, the gas hydrates melt, and everyone dies. And they all have to decide at one day. Or one could posit a planet Linearium populated entirely by people that really love skydiving, who would want to skydive everyday but that would raise the global temperature by 100 Celsius, and they'd rather be alive than skydive every day and boil to death. They opt to skydive at their birthdays at the expense of 0.3 degree global temperature rise, which each one of them finds to be an acceptable price to pay for getting to skydive at your birthday.