A utility function is like a program in a Turing-complete language. If the behaviour can be computed at all, it can be computed by a utility function.

Tim, I've seen you state this before, but it's simply wrong. A utility function is not like a Turing-complete language. It imposes rather strong constraints on possible behavior.

Consider a program which when given the choices (A,B) outputs A. If you reset it and give it choices (B,C) it outputs B. If you reset it again and give it choices (C,A) it outputs C. The behavior of this program cannot be reproduced by a utility function.

Here's another example: When given (A,B) a program outputs "indifferent". When given (equal chance of A or B, A, B) it outputs "equal chance of A or B". This is also not allowed by EU maximization.

If the aliens' wetware (er, crystalware) is so efficient that their children are already sentient when they are still tiny relative to adults, why don't the adults have bigger brains and be much more intelligent than humans? Given that they also place high values on science and rationality, had invented agriculture long before humans did, and haven't fought any destructive wars recently, it makes no sense that they have a lower level of technology than humans at this point.

Other than that, I think the story is not implausible. The basic lesson here is the same as in Robin's upload scenarios: when sentience is really cheap, no one will be valued (much) just for being sentient. If we want people to be valued just for being sentient, either the wetware/crystalware/hardware can't be too efficient, or we need to impose some kind of artificial scarcity on sentience.

Maybe we don't mean the same thing by boredom?

I'm using Eliezer's definition: a desire not to do the same thing over and over again. For a creature with roughly human-level brain power, doing the same thing over and over again likely means it's stuck in a local optimum of some sort.

Genome equivalents which don't generate terminally valued individual identity in the minds they descrive should outperform those that do.

I don't understand this. Please elaborate.

Why not just direct expected utility? Pain and pleasure are easy to find but don't work nearly as well.

I suppose you mean why not value external referents directly instead of indirectly through pain and pleasure. As long as wireheading isn't possible, I don't see why the latter wouldn't work just as well as the former in many cases. Also, the ability to directly value external referents depends on a complex cognitive structure to assess external states, which may be more vulnerable in some situations to external manipulation (i.e. unfriendly persuasion or parasitic memes) than hard-wired pain and pleasure, although the reverse is probably true in other situations. It seems likely that evolution would come up with both.

Define sexual. Most sexual creatures are too simple to value the first two. Most plausible posthumans aren't sexual in a traditional sense.

I mean reproduction where more than one party contributes genetic material and/or parental resources. Even simple sexual creatures probably have some notion of beauty and/or status to help attract/select mates, but for the simplest perhaps "instinct" would be a better word than "value".

- likely values for intelligent creatures with sexual reproduction (music, art, literature, humor)

Disagree.

These all help signal fitness and attract mates. Certainly not all intelligent creatures with sexual reproduction will value exactly music, art, literature, and humor, but it seems likely they will have values that perform the equivalent functions.

We can sort the values evolution gave us into the following categories (not necessarily exhaustive). Note that only the first category of values is likely to be preserved without special effort, if Eliezer is right and our future is dominated by singleton FOOM scenarios. But many other values are likely to survive naturally in alternative futures.

- likely values for all intelligent beings and optimization processes (power, resources) - likely values for creatures with roughly human-level brain power (boredom, knowledge) - likely values for all creatures under evolutionary competition (reproduction, survival, family/clan/tribe) - likely values for creatures under evolutionary competition who cannot copy their minds (individual identity, fear of personal death) - likely values for creatures under evolutionary competition who cannot wirehead (pain, pleasure) - likely values for creatures with sexual reproduction (beauty, status, sex) - likely values for intelligent creatures with sexual reproduction (music, art, literature, humor) - likely values for intelligent creatures who cannot directly prove their beliefs (honesty, reputation, piety) - values caused by idiosyncratic environmental characteristics (salt, sugar) - values caused by random genetic/memetic drift and co-evolution (Mozart, Britney Spears, female breasts, devotion to specific religions)

The above probably isn't controversial, rather the disagreement is mainly on the following:

- the probabilities of various future scenarios - which values, if any, can be preserved using approaches such as FAI - which values, if any, we should try to preserve

I agree with Roko that Eliezer has made his case in an impressive fashion, but it seems that many of us are still not convinced on these three key points.

Take the last one. I agree with those who say that human values do not form a consistent and coherent whole. Another way of saying this is that human beings are not expected utility maximizers, not as individuals and certainly not as societies. Nor do most of us desire to become expected utility maximizers. Even amongst the readership of this blog, where one might logically expect to find the world's largest collection of EU-maximizer wannabes, few have expressed this desire. But there is no principled way to derive an utility function from something that is not an expected utility maximizer!

Is there any justification for trying to create an expected utility maximizer that will forever have power over everyone else, whose utility function is derived using a more or less arbitrary method from the incoherent values of those who happen to live in the present? That is, besides the argument that it is the only feasible alternative to a null future. Many of us are not convinced of this, neither the "only" nor the "feasible".

Why make science a secret, instead of inventing new worlds with new science for people to explore? Have you heard of "Theorycraft"? It's science applied to the World of Warcraft, and for some, Theorycraft is as much fun as the game it's based on.

Is there something special about the science of base-level reality that makes it especially fun to explore and discover? I think the answer is yes, but only if it hasn't already been explored and then covered up again and made into a game. It's the difference between a natural and an artificial challenge.

One day we'll discover the means to quickly communicate insights from one individual to another, say by directly copying and integrating the relevant neural circuitry. Then, in order for an insight to be Fun, it will have to be novel to transhumanity, not just the person learning or discovering it. Learning something the fast efficient way will not be Fun because there's not true effort. Pretending that the new way doesn't exist, and learning the old-fashioned way, will not be Fun because there's not true victory.

I'm not sure there are enough natural problems in the universe to supply the whole of transhumanity with an adequate quantity of potential insights. "Natural" meaning not invented for the sole purpose of providing an artificial challenge. Personally, I can't see how solving the n-th random irrelevant mathematical problem is any better than lathing the n-th table leg.

Anna Salamon wrote: Is it any safer to think ourselves about how to extend our adaptation-executer preferences than to program an AI to figure out what conclusions we would come to, if we did think a long time?

First, I don't know that "think about how to extend our adaptation-executer preferences" is the right thing to do. It's not clear why we should extend our adaptation-executer preferences, especially given the difficulties involved. I'd backtrack to "think about what we should want".

Putting that aside, the reason that I prefer we do it ourselves is that we don't know how to get an AI to do something like this, except through opaque methods that can't be understood or debugged. I imagine the programmer telling the AI "Stop, I think that's a bug." and the AI responding with "How would you know?"

g wrote: Wei Dai, singleton-to-competition is perfectly possible, if the singleton decides it would like company.

In that case the singleton might invent a game called "Competition", with rules decided by itself. Anti-prediction says that it's pretty unlikely those rules would happen to coincide with the rules of base-level reality, so base-level reality would still be controlled by the singleton.

Robin wrote: Having to have an answer now when it seems an likely problem is very expensive.

(I think you meant to write "unlikely" here instead of "likely".)

Robin, what is your probability that eventually humanity will evolve into a singleton (i.e., not necessarily through Eliezer's FOOM scenario)? It seems to me that competition is likely to be unstable, whereas a singleton by definition is. Competition can evolve into a singleton, but not vice versa. Given that negentropy increases as mass squared, most competitors have to remain in the center, and the possibility of a singleton emerging there can't ever be completely closed off. BTW, a singleton might emerge from voluntary mergers, not just one competitor "winning" and "taking over".

Another reason to try to answer now, instead of later, is that coming up with a good answer would persuade more people to work towards a singleton, so it's not just a matter of planning for a contingency.

An expected utility maximizer would know exactly what to do with unlimited power. Why do we have to think so hard about it? The obvious answer is that we are adaptation executioners, not utility maximizers, and we don't have an adaptation for dealing with unlimited power. We could try to extrapolate an utility function from our adaptations, but given that those adaptations deal only with a limited set of circumstances, we'll end up with an infinite set of possible utility functions for each person. What to do?

James D. Miller: But about 100 people die every minute!

Peter Norvig: Refusing to act is like refusing to allow time to pass.

What about acting to stop time? Preserve Earth at 0 kelvin. Gather all matter/energy/negentropy in the rest of the universe into secure storage. Then you have as much time as you want to think.

Maybe we don't need to preserve all of the incompressible idiosyncrasies in human morality. Considering that individuals in the post-Singularity world will have many orders of magnitude more power than they do today, what really matter are the values that best scale with power. Anything that scales logarithmically for example will be lost in the noise compared to values that scale linearly. Even if we can't understand all of human morality, maybe we will be able to understand the most important parts.

Just throwing away parts of one's utility function seems bad. That can't be optimal right? Well, as Peter de Blanc pointed out, it can be if there is no feasible alternative that improves expected utility under the original function. We should be willing to lose our unimportant values to avoid or reduce even a small probability of losing the most important ones. With CEV, we're supposed to implement it with the help of an AI that's not already Friendly, and if we don't get it exactly right on the first try, we can't preserve even our most important values. Given that we don't know how to safely get an unFriendly AI to do anything, much less something this complicated, the probability of failure seems quite large.