Things may think very differently, but we only have to really worry about them if they think correctly, or close to it. (at our best, we think close to correctly.)

We can reason that the shortest distance between two points is a straight line. This also happens to fit our intuitions, which have nothing to do with our logic. Another mind could have other intuitions that also approximate the right answer. But if all the rituals of thought they have make them treat some curve as the shortest path, then we would beat them in a race.

If anything doesn't think perfectly rationally, then it has some cognitive hole which we might exploit. IF we can understand it, and avoid our own errors.

james andrix: we have to worry about what other Optimizers want, not just if they "think correctly". Evolution still manages to routinely defeat us without being able to think at all.

This seems relevant to the Pascal discussion, as various interlocutors try to argue with Unknown's claim that negative (egoist) consequentialist reasoning leads conveniently to psychologically comforting Christianity. While I would say that Unknown's position reflects this bias, it seems similar wishful thinking (that maximizing consequentialism will match the current values of much of OB's atheist readership) plays a role in many of the dismissals of Pascalian arguments. Despite making a number of arguments that negative utilitarianism or negative egoism do not lead to Unknown's conclusion, I remain especially wary of claims that a consequentialist would produce intuitively acceptable outcomes rather than dismantling the universe looking for the Dark Lords of the Matrix or magic physics to generate great utility.

Larry D'Anna: we have to worry about what other Optimizers want, not just if they "think correctly".

I argue that if there exist objective values which are implicit in the structure of our universe, then some significant fraction of possible minds will approximate those objective values.

However, those objective values probably differ quite a lot from most of what most human beings find important in their lives; for example our obsessions with sex, romance and child-rearing probably aren't in there.

Having re-read Eliezer's work on "the bottom line", and this piece on optimism, I am re-assesing very carefully what I think those objective values might look like. I am going to try very hard to make sure I don't simply rationalize those things that I have already decided (or been genetically programmed to think) are valuable.

Is there a good way to overcome a bias - such as the above - that one is acutely aware of?

"And because we can more persuasively argue, for what we honestly believe, we have evolved an instinct to honestly believe that other people's goals, and our tribe's moral code, truly do imply that they should do things our way for their benefit."

Great post overall, but I'm skeptical of this often-repeated element in OB posts and comments. I'm not sure honest believers always, or even usually, have a persuasion advantage. This reminds me of some of Michael Vassar's criticism of nerds thinking of everyone else as a defective nerd (nerd defined as people who value truth-telling/sincerity over more political/tactful forms of communication.

I remain especially wary of claims that a consequentialist would produce intuitively acceptable outcomes rather than dismantling the universe looking for the Dark Lords of the Matrix or magic physics to generate great utility.

Me too.

Eliezer.... This post terrifies me. How on earth can humans overcome this problem? Everyone is tainted. Every group is tainted. It seems almost fundamentally insurmountable...

And yet somehow, bridges stay up.

So you go in search of a mathematics of FAI that produces precision-grade predictions.

However, you're correct that all vaguely optimistic arguments produced by various meddling dabblers about how friendly they expect their AI design to be, can be tossed out the window.

It's not a challenge we have the option of avoiding, either.

Carl:"Unknown's [sic] claim that negative (egoist) consequentialist reasoning leads conveniently to psychologically comforting Christianity. [...] do not lead to Unknown's [sic] conclusion [...]"

Correction: this is Utilitarian's claim.

All of this reminds me of something I read in Robert Sapolski's book "Monkey Luv" (a really fluffy pop-sci book about baboon society, though Sapolski himself in person is quite amazing), about how human populations under different living conditions had almost predictable (at least in hindsight)explicative religions. People living in rainforests with many different creatures struggling at cross-purposes to survive developed polytheistic religions in which gods routinely fought and destroyed each other for their own goals. Desert dwellers (semites) saw only one great expanse of land, one horizon, one sky, one ecosystem, and so invented monotheism.

I wonder what god(s) we 21st century American rationalists will invent...

A degenerated Singularitarianism would fit the bill.

Roko: What would it even mean for an objective value to be implicit in the structure of the universe? I'm having a hard time imagining any physical situation where that would even make sense. And even if it did, it would still be you that decides to follow that value. Surely if you discovered an objective value implicit in the structure of the universe that commanded you to torture kittens, you would ignore it.

Z.M. Davis,

Occam's razor makes me doubt that we have two theoretical negative utilitarians (with egoistic practice) who endorse Pascal's wager, with similar writing styles and concerns, bearing Internet handles that begin with 'U.'

Larry D'Anna: Yes, if the agent is powerful you might be killed even by mindless thrashings.

My point was that in an adversarial situation, you should assume your opponent will always make perfect choices. Then their mistakes are to your advantage. If you're ready for optimal thrashings, random thrashings will be easier. If you know your opponent will make some kind of mistake, then you can step a trap. But for that you have to understand it very well.

If we are up against an AI that always makes the right choice, then it will almost certainly get what it wants.

If we're going to say that evolution doesn't think, then we should also say it doesn't want. Arguably wanting is anthropomorphizing.

I think we can still use the plan-for-the-strogest-possible-attack strategy though, because we can define it in terms of what we want. (but we still have to look at how the opponent 'thinks', not how we do)

Roko: then some significant fraction of possible minds will approximate those objective values.

I would say that effective minds would find those values as useful strategies. (and for this reason evolved minds would approximate them.)

But all possible minds would be all over the place.

And yet, you keep using computation metaphors to talk about brains: "preference ordering", "solution space", "relative to goals", and so on. If you are criticizing a given attitude, shouldn't you avoid it? And given your own biases, should you really be so demeaning to the collective-evolution proponents? Sorry, but it smells like you are predicting the downfall of their theories after the fact.

However, both Unknown and Utilitarian claim to be distinct, and I won't contest their claims.

Marcio: Computation is a generalization that extends to the brain. The brain is not a generalization that extends to computational processes. This is not a two-way street.

Humans faced with resource constraints did find the other approach.

Traditionally, rather than restrict our own breeding, our response has been to enslave our neighbors. Force them to work for us, to provide resources for our own children. But don't let them have children. Maybe castrate the males, if necessary kill the female's children. (It was customary to expose deformed or surplus children. If a slave does get pregnant, who's child is surplus?)

China tried the "everybody limit their children" approach. Urban couples were allowed one child, farm couples could have two. Why the difference? China officially did not have an "other" to enslave. They had to try to make it fair. But why would the strong be fair to the weak? Slaves are bred when there's so much room to expand that the masters' children can't fill the space, and another requirement is that it's easier, cheaper, or safer to breed them than to capture more.

Traditionally slavery was the humane alternative to genocide.

Why didn't Eliezer think that way? My guess is that he is a good man and so he supposed that human populations would think in terms of what's good and fair for everyone, the way he does.

He applied anthropomorphic optimism to human beings.

Great comments thread! Thanks all!

Seconding Roko, Carl, HA, Nick T, etc.

Eliezer or Robin: Can you cite evidence for "we can more persuasively argue, for what we honestly believe". My impression is that it has been widely assumed in evolutionary psychology and fairly soundly refuted in the general psychology of deception, which tells us that the large majority of people detect lies at about chance and that similar effort seems to enable the development of the fairly rare skill of the detection of lies and evasion of such detection.

Carl: Unknown and Utilitarian could be distinct but highly correlated (we're both here after all). In principle we could see them as both unpacking the implications of some fairly simple algorithm. Have you noticed them both making the same set of mistakes in their efforts to understand Bayesian reasoning, anthropics, decision theory, etc? Still could be the same program running on different sets of wetware.

"However, those objective values probably differ quite a lot from most of what most human beings find important in their lives; for example our obsessions with sex, romance and child-rearing probably aren't in there."

Several years ago, I was attracted to pure libertarianism as a possible objective morality for precisely this reason. The idea that, eg., chocolate tastes good can't possibly be represented directly in an objective morality, as chocolate is unique to Earth and objective moralities need to apply everywhere. However, the idea of immorality stemming from violation of another person's liberty seemed simple enough to arise spontaneously from the mathematics of utility functions.

It turns out that you do get a morality out of the mathematics of utility functions (sort of), in the sense that utility functions will tend towards certain actions and away from others unless some special conditions are met. Unfortunately, these actions aren't very Friendly; they involve things like turning the universe into computronium to solve the Riemann Hypothesis (see http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf for some examples). If libertarianism really was a universal morality, Friendly AI would be much simpler, as we could fail on the first try without the UFAI killing us all.

Michael: both making the same set of mistakes in their efforts to understand Bayesian reasoning, anthropics, decision theory, etc?

Would you like to elaborate on what those mistakes are (thereby helping out this simple but misguided algorithm)? If there are links to existing sources, that would be enough.

Vassar, surprised to see you seconding Roko. Did you confuse his post with someone else's?

Roko, convergent instrumental values are neither universalizable nor objective. A paperclip maximizer and a cheesecake maximizer both want to survive, and to capture all the resources in the universe; they don't want each other to survive, and would disagree about how to allocate the resources. Even if they negotiate, they would still prefer the other dead; and we humans would prefer that both of them never exist in the first place.

J Thomas, I didn't say that no human wouldn't think of cannibalizing girls, I'm saying it wouldn't occur to you, and did not in fact occur to the group selectionists. Anthropomorphism can just as easily be 1st-World-o-morphism and usually is.

The "mistake" Michael is talking about it the belief that utility maximization can lead to counter intuitive actions, in particular actions that humanly speaking are bound to be useless, such as accepting a Wager or a Mugging.

This is in fact not a mistake at all, but a simple fact (as Carl Shulman and Nick Tarleton suspect.) The belief that it does not is simply a result of Anthropomorphic Optimism as Eliezer describes it; i.e. "This particular optimization process, especially because it satisfies certain criteria of rationality, must come to the same conclusions I do." Have you ever considered the possibility that your conclusions do not satisfy those criteria of rationality?

Lara Foster: We know the location and connections of every one of C. elegans 213 neurons... Why can't we make a device that will do the same thing yet? Too much anthropomorphism?

Too much anthropomorphism is precisely what a lot of AI research looks like to me (Google ["cognitive architecture"] for the sort of thing I mean), although C. elegans isn't a good example of what we can't do. All it takes to climb concentration gradients is a biased random walk, and light-seeking is a standard project with hobby robot kits.

Building it the size of a caterpillar, let alone a flatworm, is a more challenging task, but not one of AI. (However, I would not be surprised if some AI researcher were to claim that smallness is the key missing ingredient, as has been claimed in the past for computational power, parallelism, analog computation, emotions, and embodiment.)

We don't know how brains work, and our view from the inside doesn't tell us.

Carl Shulman: Occam's razor makes me doubt that we have two theoretical negative utilitarians (with egoistic practice) who endorse Pascal's wager, with similar writing styles and concerns, bearing Internet handles that begin with 'U.' michael vassar: Unknown and Utilitarian could be distinct but highly correlated (we're both here after all). In principle we could see them as both unpacking the implications of some fairly simple algorithm.

With thousands of frequent-poster-pairs with many potentially matchable properties, I'm not too shocked to find a pair that match on six mostly correlated properties.

Kennaway- I meant why can't we make something that C. elegans does in the same way that C. elegans does it using it's neural information. Clearly our knowledge must be incomplete in some respect. If we could do that, then imitating not only the size, but the programming of the caterpillar would be much more feasible. At least three complex programs are obvious: 1) crawl -coordinated and changeable sinusoidal motion seems a great way to move, yet the MIT 'caterpillar' is quite laughable in comparison to the dexterity of the real thing, 2)Seek- this involves a circular motion of the head, sensing some chemical signal, and changing directions accordingly, 3) Navigate- caterpillar is skillfully able to go over, under, and around objects, correcting its path to its original without doing the weird head-twirling thing, indicating that aside from chemeoattraction, it has some sense of directional orientation, which it must have or else its motion would be a random walk with correction and not a direct march. I wonder how much of these behaviors operate independently of the brain.

Lara Foster, what do you put as the odds that our knowledge is complete, but no one has tried to make something directly copying C elegans?

From Nick Bostrom's paper on infinite ethics:

"If there is an act such that one believed that, conditional on one’s performing it, the world had a 0.00000000000001% greater probability of containing infinite good than it would otherwise have (and the act has no offsetting effect on the probability of an infinite bad), then according to EDR one ought to do it even if it had the certain side‐effect of laying to waste a million human species in a galactic‐scale calamity. This stupendous sacrifice would be judged morally right even though it was practically certain to achieve no good. We are confronted here with what we may term the fanaticism problem."

Later:

"Aggregative consequentialism is often criticized for being too “coldly numerical” or too revisionist of common morality even in the more familiar finite context. Suppose that I know that a certain course of action, though much less desirable in every other respect than an available alternative, offers a one‐in‐a‐million chance of avoiding catastrophe involving x people, where x is finite. Whatever else is at stake, this possibility will overwhelm my calculations so long as x is large enough. Even in the finite case, therefore, we might fear that speculations about low‐probability‐high‐stakes scenarios will come to dominate our moral decision making if we follow aggregative consequentialism."

Exactly. Utility maximizing together with an unbounded utility function necessarily lead to what Nick calls fanaticism. This is the usual use of the term: people call other people fanatics when their utility functions seem to be unbounded.

As Eliezer has pointed out, it is a dangerous sign when many people agree that something is wrong without agreeing why; we see this happening in the case of Pascal's Wager and Pascal's Mugging. In reality, a utility maximizer with an unbounded utility function would accept both. The readers of this blog, being human, are not utility maximizers. But they are unwilling to admit it because certain criteria of rationality seem to require being such.

My point was that in an adversarial situation, you should assume your opponent will always make perfect choices. Then their mistakes are to your advantage. If you're ready for optimal thrashings, random thrashings will be easier.

It isn't that simple. When their perfect choice mean you lose, then you might as well hope they make mistakes. Don't plan for the worst that can happen, plan for the worst that can happen which you can still overcome.

One possible mistake they can make is to just be slow. If you can hit them hard before they can react, you might hurt them enough to get a significant advantage. Then if you keep hitting them hard and fast they might reach the point they're trying to defend against the worst you can do. While they are trying to prepare for the worst attack you can make, you hit them hard with the second-worst attack that they aren't prepared for. Then when they try to defend against whatever they think you'll do next, you do something else bad. It's really ideal when you can get your enemy into that stance.

Of course it's even better when you can persuade them not to be your enemy in the first place. If you coudl do that reliably then you would do very well. But it's hard to do that every time. The enemy gets a vote.

"If there is an act such that one believed that, conditional on one’s performing it, the world had a 0.00000000000001% greater probability of containing infinite good than it would otherwise have (and the act has no offsetting effect on the probability of an infinite bad), then according to EDR one ought to do it even if it had the certain side‐effect of laying to waste a million human species in a galactic‐scale calamity.

The assumption is that when you lay waste to a million human species the bad that is done is finite.

Is there solid evidence for that? If there's any slightest chance that it will result in infinite bad, then the problem is much more complicated.

There has to be not the 0.00000000000001% probability that your evil might be infinite, before this reasoning makes sense.

Doug- too much stuff to put into an actual calculation, but I doubt we have complete knowledge, given how little we understand epigentics (iRNA, 22URNAs, and other micro RNAs), synaptic transcription, cytoskeletal transport, microglial roles, the enteric nervous system, native neuro-regeneration, and low and behold, neurotransmitters themselves. The 3rd edition of Kandel I was taught out of as an undergrad said nothing of orexins, histamine, the other roles of meletonin beyond the pineal gland, or the functions of the multifarious set of cannibinoid receptors, yet we now know (a short 2 years later) that all of these transmitters seem to play critical roles. Now, not being an elegans gal, I don't know if it has much simpler neurotransmission than we do. I would venture to guess it is simpler, but not extraordinarily so, and probably much more affected by simple epigenetic mechanisms like RNA interference. In C. elegans, iRNA messages are rapidly amplified, quickly shutting off the target gene in all of its cells (mammals have not been observed to amplify). Now, here's the kicker- it gets into the germ cells too! So offspring will also produce iRNAs and shut off genes! Now, due to amplification error, iRNAs are eventually lost if the worms are bred long enough, but Craig Mellow is now exploring the hypothesis that amplified iRNAs can be eventually permenantly disable (viral) DNA that's been incorporated into the C. elegans genome, either by more permenant epigenetic modification (methylation or coiling), or by splicing it out... Sorry for the MoBio lecture, but DUDE! This stuff is supercool!!!

Actually, if you want a more serious answer to your question, you should contact Sydney Brenner or Marty Chalfie, who actually worked on the C. elegans projects. Brenner is very odd and very busy, but Chalfie might give you the time of day if you make him feel important and buy him lunch.... Marty is an arrogant sonuvabitch. Wouldn't give me a med school rec, because he claimed not to know anything about me other than that I was the top score in his genetics class. I was all like, "Dude! I was the one who was always asking questions!" And he said, "Yes, and then class would go overtime." Lazy-Ass-Sonuvabitch... But still a genius.

If there's any slightest chance that it will result in infinite bad, then the problem is much more complicated.

There's always a nonzero chance that any action will cause an infinite bad. Also an infinite good. Even with finite but unbounded utility functions, this divergence occurs.

Utility maximizing together with an unbounded utility function necessarily lead to what Nick calls fanaticism.

Bounded utility functions have counterintuitive results as well. Most of these only show up in rare (but still realistic) global "what sort of world should we create" situations, but there can be local effects too; as I believe Carl Shulman pointed out, bounded utility causes your decisions to be dominated by low-probability hypotheses that there are few people (so your actions can have a large effect.)

Nick, can you explain how that happens with bounded utility functions? I was thinking basically something like this: if your maximum utility is 1000, then something that has a probability of one in a million can't have a high expected value or disvalue, because it can't be multiplied by more than 1000, and so the expected value can't be more than 0.001.

This seems to me the way humans naturally think, and the reason that sufficiently low-probability events are simply ignored.

My argument for bounded utility is this: if a mind is so far into the future that specifying where it is takes as much information input as generating the mind from scratch, in what sense is it more present-in-the-world than the mind-from-scratch is?

@ Unknown: "The readers of this blog, being human, are not utility maximizers. But they are unwilling to admit it because certain criteria of rationality seem to require being such."

• I am proud to admit that I am not a utility maximizer. I'm actually busy writing a blog post on why utility maximization is a misguided ethical theory.

@Eliezer: Vassar, surprised to see you seconding Roko. Did you confuse his post with someone else's? Roko, convergent instrumental values are neither universalizable nor objective.

• What exactly do you mean by these words? I fear that we may be using the same words to mean different things, hence our disagreements...

A paperclip maximizer and a cheesecake maximizer both want to survive, and to capture all the resources in the universe; they don't want each other to survive, and would disagree about how to allocate the resources. Even if they negotiate, they would still prefer the other dead;

• I agree with this, but why do you think that this is important?

and we humans would prefer that both of them never exist in the first place.

• I agree with this, I don't advocate creating a paperclip maximizer.

@Tom McCabe: It turns out that you do get a morality out of the mathematics of utility functions (sort of), in the sense that utility functions will tend towards certain actions and away from others unless some special conditions are met. Unfortunately, these actions aren't very Friendly; they involve things like turning the universe into computronium to solve the Riemann Hypothesis (see http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf for some examples).

• I'm not actually focusing on the values/ethics/morality that you can get out of utility functions, I'm asking the more general question of what values/ethics/morality you can get out of the mathematics of an agent with goals interacting with an environment. Utilitarian agents are just one example of such agents.

I think that the canonical set of instrumental values that Omohundro, Hollerith and myself have been talking about have perhaps been slated more than they deserve. To me, it seems that the four "basic drives" - Self-preservation, Acquisition, Efficiency, Creativity, embody precisely the best aspects of human civilization. Not the best aspects of individual behavior, mind you, which is a different problem.

But I think that we would all like to see a civilization that acquired more free energy and space, worked harder to preserve its own existence (I think some people at oxford might have had a small gathering about that one recently), used its resources more efficiently, and strove for a greater degree of creativity. In fact I cannot think of a more concise and general description of the goals of transhumanism than Omohundro's basic AI drives, where the "agent" is our entire human civilization.

"Nick, can you explain how that happens with bounded utility functions? I was thinking basically something like this: if your maximum utility is 1000, then something that has a probability of one in a million can't have a high expected value or disvalue, because it can't be multiplied by more than 1000, and so the expected value can't be more than 0.001." Suppose that your utility function is U=1-1/X where X is the number of cheesecakes produced, so your utility is bounded between 0 and 1. Quantum physics, modal realism, and other 'Big World' theories indicate that X approaches infinity. Therefore, conditional on the world being anything like it appears to be, the utility of any action is infinitesimal. If there is a one in a million subjective probability that no cheesecakes exist and you can create some (because of weird, probably false physical theories, claims about the Dark Lords of the Matrix, etc) then the expected value of pursuing the routes conditional on those weird theories will be ~1/1,000,000, absurdly greater than any course of action conditional on 'normal' understanding.

Since we don't currently have utility functions, should we assign vastly greater weight to scenarios where our values renormalize to something resembling an unbounded utility function than a bounded one?

steven: is there a similar spatial argument to bound the utility attainable at any given time?

BTW, I'm disappointed Eliezer hasn't addressed "...a consequentialist would produce intuitively acceptable outcomes rather than dismantling the universe looking for the Dark Lords of the Matrix or magic physics to generate great utility."

Carl, that's not a feature of all bounded utility functions. If you discount exponentially for time (or more plausibly for the algorithmic complexity of locating an observer-moment in the universe, and I might want to do this for reasons independent of Pascal) then the value of something does not depend on how much of that something is already in existence, but it still seems like you couldn't get to infinite utilities.

Nick, it seems like if you discounted for algorithmic complexity that should imply some way to discount for extreme distances in space, yes. Not sure how exactly.

There's always a nonzero chance that any action will cause an infinite bad. Also an infinite good.

Then how can you put error bounds on your estimate of your utility function?

If you say "I want to do the bestest for the mostest, so that's what I'll try to do" then that's a fine goal. When you say "The reason I killed 500 million people was that according to my calculations it will do more good than harm, but I have absolutely no way to tell how correct my calculations are" then maybe something is wrong?

Roko: it's good to see that there is at least one other human being here.

Carl, thanks for that answer, that makes sense. But actually I suspect that normal humans have bounded utility functions that do not increase indefinitely with, for example, cheese-cakes. Instead, their functions have an absolute maximum which is actually reachable, and nothing else that is done will actually increase it.

Michael Vassar: Actually in real life I do some EXTREMELY counterintuitive things. Also, I would be happy to know the actual consequences of my beliefs. I'm not afraid that I would have to act in any particular way, because I am quite aware that I am a human being and do not have to act according to the consequences of my beliefs unless I want to. I often hold beliefs without acting on them, in fact.

If there is a 90% chance that utility maximization is correct, and a 10% chance that Roko is correct (my approximate estimates), how should one act? You cannot simply "use the math", as you suggest, because conditional on the 10% chance, you shouldn't be using the math at all.

Michael- ah yes, that makes a lot of sense. Of course if the worm's only got 213 neurons, it's not going to have hundreds of neurotransmitters. That being said, it might have quite a few different receptor sub-types and synaptic modification mechanisms. Even so... It would seem theoretically feasible to me for someone to hook up electrodes to each neuron at a time and catalog not only the location and connections of each neuron, but also what the output of each synapse is and what the resulting PSPs are during normal C. elegans behaviors... Now that's something I should tell Brenner about, given his penchant for megalomaniacal information gathering projects (he did the C. Elegans genome, a catalog of each cell in its body throughout its embryonic development, and its neural connections).

But later on, Michael J. Wade went out and actually created in the laboratory the nigh-impossible conditions for group selection.  Wade repeatedly selected insect subpopulations for low population numbers.  Did the insects evolve to restrain their breeding, and live in quiet peace with enough food for all, as the group selectionists had envisioned?

No; the adults adapted to cannibalize eggs and larvae, especially female larvae.

What would have happened if Wade had also repeatedly selected subpopulations for not doing that?

Such a low-ranking solution as "Everyone have as many kids as possible, then cannibalize the girls" would not be generated in your search process.

Like...  "A Modest Proposal"?  I would suggest that low-ranking solutions are very often generated and are simply discarded without comment in the vast majority of cases.  The only way "efficiency" enters into it comes from the way we start our search for solutions by considering how to adapt already known solutions to similar problems.

This does, in fact, show up in evolution as well.  Adapting existing solutions is far more common than inventing something entirely new.  Like how the giraffe has the same number of cervical vertebra as (so far as I know) every other four-legged mammal on the planet.