Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Leaky Generalizations

20 Post author: Eliezer_Yudkowsky 22 November 2007 09:16PM

Followup toTerminal Values and Instrumental Values

Are apples good to eat?  Usually, but some apples are rotten.

Do humans have ten fingers?  Most of us do, but plenty of people have lost a finger and nonetheless qualify as "human".

Unless you descend to a level of description far below any macroscopic object - below societies, below people, below fingers, below tendon and bone, below cells, all the way down to particles and fields where the laws are truly universal - then practically every generalization you use in the real world will be leaky.

(Though there may, of course, be some exceptions to the above rule...)

Mostly, the way you deal with leaky generalizations is that, well, you just have to deal.  If the cookie market almost always closes at 10pm, except on Thanksgiving it closes at 6pm, and today happens to be National Native American Genocide Day, you'd better show up before 6pm or you won't get a cookie.

Our ability to manipulate leaky generalizations is opposed by need for closure, the degree to which we want to say once and for all that humans have fingers, and get frustrated when we have to tolerate continued ambiguity.  Raising the value of the stakes can increase need for closure - which shuts down complexity tolerance when complexity tolerance is most needed.

Life would be complicated even if the things we wanted were simple (they aren't).  The leakyness of leaky generalizations about what-to-do-next would leak in from the leaky structure of the real world.  Or to put it another way:

Instrumental values often have no specification which is both compact and local.

Suppose there's a box containing a million dollars.  The box is locked, not with an ordinary combination lock, but with a dozen keys controlling a machine that can open the box.  If you know how the machine works, you can deduce which sequences of key-presses will open the box.  There's more than one key sequence that can trigger the button.  But if you press a sufficiently wrong sequence, the machine incinerates the money.  And if you don't know about the machine, there's no simple rules like "Pressing any key three times opens the box" or "Pressing five different keys with no repetitions incinerates the money."

There's a compact nonlocal specification of which keys you want to press:  You want to press keys such that they open the box. You can write a compact computer program that computes which key sequences are good, bad or neutral, but the computer program will need to describe the machine, not just the keys themselves.

There's likewise a local noncompact specification of which keys to press: a giant lookup table of the results for each possible key sequence.  It's a very large computer program, but it makes no mention of anything except the keys.

But there's no way to describe which key sequences are good, bad, or neutral, which is both simple and phrased only in terms of the keys themselves.

It may be even worse if there are tempting local generalizations which turn out to be leaky.  Pressing most keys three times in a row will open the box, but there's a particular key that incinerates the money if you press it just once.  You might think you had found a perfect generalization - a locally describable class of sequences that always opened the box - when you had merely failed to visualize all the possible paths of the machine, or failed to value all the side effects.

The machine represents the complexity of the real world.  The openness of the box (which is good) and the incinerator (which is bad) represent the thousand shards of desire that make up our terminal values.  The keys represent the actions and policies and strategies available to us.

When you consider how many different ways we value outcomes, and how complicated are the paths we take to get there, it's a wonder that there exists any such thing as helpful ethical advice.  (Of which the strangest of all advices, and yet still helpful, is that "The end does not justify the means.")

But conversely, the complicatedness of action need not say anything about the complexity of goals.  You often find people who smile wisely, and say, "Well, morality is complicated, you know, female circumcision is right in one culture and wrong in another, it's not always a bad thing to torture people.  How naive you are, how full of need for closure, that you think there are any simple rules."

You can say, unconditionally and flatly, that killing anyone is a huge dose of negative terminal utility.  Yes, even Hitler.  That doesn't mean you shouldn't shoot Hitler.  It means that the net instrumental utility of shooting Hitler carries a giant dose of negative utility from Hitler's death, and an hugely larger dose of positive utility from all the other lives that would be saved as a consequence.

Many commit the type error that I warned against in Terminal Values and Instrumental Values, and think that if the net consequential expected utility of Hitler's death is conceded to be positive, then the immediate local terminal utility must also be positive, meaning that the moral principle "Death is always a bad thing" is itself a leaky generalization.  But this is double counting, with utilities instead of probabilities; you're setting up a resonance between the expected utility and the utility, instead of a one-way flow from utility to expected utility.

Or maybe it's just the urge toward a one-sided policy debate: the best policy must have no drawbacks.

In my moral philosophy, the local negative utility of Hitler's death is stable, no matter what happens to the external consequences and hence to the expected utility.

Of course, you can set up a moral argument that it's an inherently a good thing to punish evil people, even with capital punishment for sufficiently evil people.  But you can't carry this moral argument by pointing out that the consequence of shooting a man with a leveled gun may be to save other lives.  This is appealing to the value of life, not appealing to the value of death.  If expected utilities are leaky and complicated, it doesn't mean that utilities must be leaky and complicated as well.  They might be!  But it would be a separate argument.

Comments (28)

Sort By: Old
Comment author: Tiiba2 23 November 2007 03:21:50AM 6 points [-]

Oh please. Sure, diseases played a role. But they weren't the only factor.

Anyway, National Turkey Genocide Day. That's for sure.

Comment author: Cyan2 23 November 2007 05:20:07AM 4 points [-]

rukidding,

The immigration you refer to took place 10,000 years ago, and (this is key!) the land was uninhabited by humans when those Asian immigrants took up residence. But whether you call them natives or Asian immigrants, European settlers slaughtered them and claimed their lands.

Jared Diamond, in his book The Third Chimpanzee, quotes several famous Americans on the subject of the native peoples. Here are three quotes:

George Washington: "The immediate objectives are the total destruction and devastations of the settlements. It will be essential to ruin their crops in the ground and prevent their planting more."

Andrew Jackson: "They have neither the intelligence, the industrym the moral habits, not the desire of improvement which are essential to any favorable change in their condition. Established in the midst of another and a superior race, and without appreciating the causes of their inferiority or seeking to control them, they must necessarily yield to the force of circumstances and ere long disappear."

Theodore Roosevelt: "The settler and pioneer have at bottom had justice on their side; this great continent could not have been kept as nothing but a game preserve for squalid savages."

The other quotes, from Benjamin Franklin, Thomas Jefferson, John Quincy Adams, James Monroe, John Marshall, William Henry Harrison, and Philip Sheridan, are just as damning.

Comment author: Cyan2 23 November 2007 05:30:57AM 0 points [-]

Erratum: George Washington said, "...their settlements...", not, "...the settlements..."

Comment author: Constant2 23 November 2007 05:45:54AM 2 points [-]

Thanksgiving is a tradition that started rather early. Wikipedia has it at about 1621 OR 1619 (depending on where you start counting). What genocides occurred on or before 1621, that Thanksgiving might have commemorated? Wikipedia says that Indians were invited to the Thanksgiving celebration of 1621. If the Thanksgiving celebration was indeed a celebration of a genocide, then it was presumably of a genocide of some other tribe.

Of course, there is also the possibility that Eliezer is trying to be flippant. Not everyone finds genocide to be funny, but some do.

Comment author: Constant2 23 November 2007 05:51:30AM -2 points [-]

Cyan - thanks for the enlightenment. As we all know, unkind words literally kill people dead. It need hardly be said that the unkind words you quote were not cherry-picked, and that there were no kind words ever said by whites about the native americans. And native americans were on their side entirely without sin, the first, last, and only people on the planet ever to treat new arrivals with open arms. They never gave whites any reason to think of them as enemies.

Comment author: Nick_Tarleton 23 November 2007 06:04:00AM 5 points [-]

Constant, Cyan said none of those things.

Could someone killthread the Native American discussion?

Comment author: Constant2 23 November 2007 06:54:24AM 1 point [-]

Nick, maybe Cyan could clarify what he was trying to imply by his statement that the words were "damning". He didn't actually draw any inferences but he sure as hell implied inferences. I pointed out what the likely inferences were and why they're bullshit.

Can someone edit the original blog entry so that an otherwise fine entry is not marred by a small bit of extremely unwise text that any semi-competent author would know would cause controversy that would swamp serious discussion of the serious point?

Comment author: Eliezer_Yudkowsky 23 November 2007 07:21:50AM 23 points [-]

I'm surprised it caused that much controversy. I was expecting people to go off on the Hitler part instead. But, hey, if that's what people want to discuss...

I'm dismayed that anyone would feel the need to defend the European colonizers as a group (apart from the many individuals who undoubtedly dealt honorably). I'm especially surprised that anyone would suggest that the genocide was okay because the various Native American tribes undoubtedly got the land in the first place by genociding the tribe that genocided the tribe that genocided the tribe... back through a few dozen genocides to whoever came over the land bridge, who probably wiped out a tribe or two on their way over. Have we forgotten that two wrongs don't make a right?

Be it far from me to spoil a holiday. Being gloomy all the time won't help anything. But I celebrate Newtonmas, not Christmas (since Newton actually was born on Dec 25th) - celebrating a holiday is one thing, celebrating a lie is another. And if there's anything to Give Thanks for, it's that in modern America, land transfers are done by contract.

I'm amazed that people want their stories to be so one-sided. If the Native Americans were bad people, it follows that the Europeans must be good? If the Europeans were bad, it follows that the Native Americans must be good? Who says the story must have heroes? There were a few good people, I'm sure. There usually are. But a team worthy of rooting for? Why would you expect any group, of a few centuries past, to live up to your own morals? Wouldn't the whole story look different if that had been true?

Let's just all agree that we won't commit genocide, and we'll have learned whatever there was to learn.

Comment author: Dragon 04 April 2012 04:19:07AM 0 points [-]

Is it really meaningful to say Newton was born on Dec 25th when he was born a fortnight after the solstice?

Comment author: Dan 23 November 2007 07:55:15AM -1 points [-]

Most apples aren't good to eat. Only those specifically bred for such purpose.

Comment author: Capla 01 June 2015 09:05:45PM 0 points [-]

Aren't most of the apples on earth precisely the ones we bread to be edible (and tasty)?

Comment author: Tiiba2 23 November 2007 08:15:27AM 4 points [-]

"I'm especially surprised that anyone would suggest that the genocide was okay"

Okay, I'm afraid I have to accuse you of attacking a strawman. rukidding said that there was no genocide.

Comment author: Doug_S. 23 November 2007 08:53:28AM 2 points [-]

Hey, we stole this land fair and square! ;)

Anyway, on "The ends don't justify the means"...

I think, in some cases, the ends clearly do justify the means. For example, killing someone is generally considered wrong, but it's generally considered to me morally permissible to kill someone in self-defense or in defense of others. If you use some "evil" means to achieve a "good" end - and you do achieve that end - then, if the magnitude of the good achieved is greater than the magnitude of the evil, the use of the evil means can often be justified. (Of course, there is always the obligation to try to find a third alternative, but that's a complication beyond the scope of my argument.)

There is a catch, though. Justifying bad means through good ends is dangerous, because people often fail to achieve the ends they were hoping for. In the infamous trolley problem, if you push the fat man onto the tracks hoping to stop the runaway trolley, but the trolley still doesn't stop, you just killed the fat man for nothing. History is filled with examples of people who resorted to evil means to achieve good ends, and failed. When you resort to evil means, you have a greater obligation to verify that you really are going to achieve a net good, because if you screw up, the consequences are much, much worse than if you refused to employ evil means in the first place. As a practical matter, "the ends don't justify the means," although not strictly true, is still a very useful heuristic for making moral decisions, because it puts a floor on the amount of damage you end up doing when you make mistakes.

Does this make any sense?

Comment author: JDM 09 June 2013 03:49:04AM *  0 points [-]

I think the statement "the end doesn't justify the means" is somewhat silly in it's own right. While it would typically be argued in the sense that killing someone to improve someone else's life is not OK, for example, would the person dying not be equally a part of the end as the other's life improving? It seems more likely to result in double counting or a similar fallacy to try to separate an action into end and means in the first place, when everything already has an impact on the end in some way.

That said, the understood meaning is not the same as its literal value, and the meaning closer to how it is understood of "consider all the consequences of your actions" does have value.

Comment author: Constant2 23 November 2007 08:58:36AM 2 points [-]

I'm especially surprised that anyone would suggest that the genocide

Look, you need to do a minimal amount of research on points you raise especially when they're controversial. And if you are surprised that they raise any controversy, then you are like the person who thought that Reagan could not possibly have won because nobody they knew voted for him. Consider this your introduction to the wider world. Here's wikipedia on the topic (I quote from it for convenience as usual, not as an authority).

In his book American Holocaust, David Stannard argues that the destruction of the aboriginal peoples of the Americas, in a "string of genocide campaigns" by Europeans and their descendants, was the most massive act of genocide in the history of the world.[13][20] While no mainstream historian denies that death and suffering were unjustly inflicted by a number of Europeans upon a great many American natives, most scholars of the subject maintain that genocide, which is a crime of intent, was not the intent of European colonization. Historian Stafford Poole wrote: "There are other terms to describe what happened in the Western Hemisphere, but genocide is not one of them. It is a good propaganda term in an age where slogans and shouting have replaced reflection and learning, but to use it in this context is to cheapen both the word itself and the appalling experiences of the Jews and Armenians, to mention but two of the major victims of this century."[21]

I come here half-expecting reflection and learning, and am disappointed when I find propaganda terms and slogans.

I'm amazed that people want their stories to be so one-sided.

Physician, heal thyself. You just got done calling Thanksgiving "National Native American Genocide Day." Where is the balance in that?

Furthermore, why pollute that particular celebration with the taint of some injustices committed by Americans? Shall we call Christmas "Forced Conversion of Jews Day", polluting it with some sins of Christians? Shall we call July 4th "Bombing of Hiroshima and Nagasaki Day", polluting it with the taint of some terrible acts committed by the US?

Comment author: Cyan2 23 November 2007 01:05:00PM 4 points [-]

Constant, a reply in brief:

"unkind words literally kill people dead"

Incitements to violence by leading citizens may plausibly be inferred to cause death. This is not usually classed as a bullshit inference.

"the unkind words you quote were... cherry-picked"

You say cherry-picked, I say representative of government policies that were actually carried out. Tomato, tomahto.

"native americans were on their side entirely without sin... never gave whites any reason to think of them as enemies."

As Nick Tarleton noted, I never made that claim.

By "damning", I meant, "worthy of condemnation as harmful, illegal, or immoral." (That's pretty much straight from the dictionary.)

Let me just add, genocide is something humans do -- everywhere, at all times in history. (Chimps too, less efficiently.) The natives were no better or worse than the settlers, only more poorly equipped.

Comment author: Recovering_irrationalist 23 November 2007 02:15:41PM 6 points [-]

@Eliezer: Each Nazis/genocide mention adds to the risk of thread derailment. Shouldn't, but does. I'd put them next to Quantum Mechanics on a things to avoid explaining things with list.

Comment author: anon8 23 November 2007 08:30:44PM 1 point [-]

A few thoughts:

-many settlers and colonies did actually purchase land from the 'natives', sometimes other tribes refused to recognize the purchase and there was conflict, sometimes other members of the selling tribe refused to recognize the deal, sometimes settlers expanded beyond the purchased boundaries.

-some of the conflict with the 'natives' was a result of agitation by the French, and other European powers against the colonies of their European rivals.

-The closest thing to a true genocidal campaign was carried out by the Republican administration after they defeated the Confederate States, continuing in their activities of corporate welfare/warfare in response to the interests of the Northern (and Midwestern) industrialists, first with the tariff and the war, then with clearing the west of natives to build the negative-value government subsidized railroads.

Comment author: Bob3 23 November 2007 08:41:22PM 1 point [-]

Doug S,

You have to use ex ante probabilities - just because the fat dude stops the trolley once in a million times doesn't make it a moral act that one time. In practice we almost never know the probabilities, which leads to the ends-means conclusion. What's interesting is how many folks are willing to, in the face of not knowing the probabilities, substitute intentions. Which has its own saying.

Comment author: Tiiba2 23 November 2007 10:39:19PM 8 points [-]

"You have to use ex ante probabilities"

This reminded me of something. Why is the outcome of a crime used to sentence criminals, rather than the likeliest outcome? A cop died while driving to a robbery (IIRC), the robber was charged with murder. Does anyone else find that stupid?

In another instance, two kids had a fight, one fainted. The assailant would be chared with assault if the victim survived, or murder otherwise. So whether he committED murder depended on future events.

I bet this has a name. Doctrine of hindsightus perfectus or something. Anyone know?

Comment author: satt 04 November 2011 10:25:39AM 3 points [-]

The eggshell skull rule sounds like an example of this.

Comment author: g 23 November 2007 11:10:21PM 2 points [-]

Tiiba, there's a nice paper on the subject by the philosopher David Lewis. Called something like "The punishment lottery". His basic idea is that you can think of our system not as punishing those criminals who get caught doing X for doing X, but as punishing those who act in way X in a way that varies somewhat at random according to the probability distributions of outcomes of X. So as soon as you attack someone you're implicitly "sentenced" to a random outcome that might be escaping unpunished, or might be conviction for murder and execution or imprisonment for life. And the severity of the eventual punishment varies to some extent with the seriousness of the crime -- the more brutally you attack someone, the more likely you are to end up with a conviction for murder rather than for actual bodily harm.

It's not clear how far this can go towards justifying the systems we currently have (and Lewis says as much), but it's pretty ingenious.

And yes, this does seem to be related to the well known "hindsight bias" whereby we tend to think actual outcomes much more predictable-in-principle from the prior evidence than they actually were. (Unless we predicted something different, of course.)

Comment author: Chris 26 November 2007 08:43:11PM 0 points [-]

'Genocide' refers to intentions rather than consequences, but it seems to me just fine to have a National Native American Genocide Day to remind us that sometimes consequences should have been taken into consideration. Even if they weren't, which of course is another question. A bit like Iraq.. (oops !! No Damn !!!! I didn't mean to say that !!!). So let's have a nice polite debate on the Instrumental Values and the Terminal Values in the Iraq war.. I've looked hard but not found any leaky generalisations in the area.....

Comment author: Rick_Smith 27 November 2007 12:12:35AM 2 points [-]

I hate to be a pedant (no really!), but most humans have eight fingers and two thumbs.

Comment author: Christophe_Poucet 28 November 2007 08:15:32PM 3 points [-]

In computer science, there is a similar concept called "Leaky abstractions"

Comment author: andybak 04 December 2012 12:26:11AM 1 point [-]

Which is a better name entirely. An abstraction is a kind of container so the idea of it leaking makes sense. How does a generalization 'leak'?

The idea was much stronger in it's original domain.

Comment author: DanielLC 29 December 2009 12:07:34AM 0 points [-]

The Pilgrims were in pretty desperate shape when they got here, and they quickly recognized that they needed to work with the Native Americans. They didn't start stealing land and everything that resulted in until the next generation, which was full of people who's lives weren't saved by the Native Americans.

Other settlements had been fighting the Native Americans for a while, but they're not who Thanksgiving celebrates.

Also, they were trying to get the Native Americans to leave. It can't really be called a genocide until America was large enough that there was nowhere to leave to.