Update 5-24-2013: A cleaned-up, citable version of this article is now available on MIRI's website.

Co-authored with crazy88

Summary: Yudkowsky's "coherent extrapolated volition" (CEV) concept shares much in common Ideal Advisor theories in moral philosophy. Does CEV fall prey to the same objections which are raised against Ideal Advisor theories? Because CEV is an epistemic rather than a metaphysical proposal, it seems that at least one family of CEV approaches (inspired by Bostrom's parliamentary model) may escape the objections raised against Ideal Advisor theories. This is not a particularly ambitious post; it mostly aims to place CEV in the context of mainstream moral philosophy.

What is of value to an agent? Maybe it's just whatever they desire. Unfortunately, our desires are often the product of ignorance or confusion. I may desire to drink from the glass on the table because I think it is water when really it is bleach. So perhaps something is of value to an agent if they would desire that thing if fully informed. But here we crash into a different problem. It might be of value for an agent who wants to go to a movie to look up the session times, but the fully informed version of the agent will not desire to do so — they are fully-informed and hence already know all the session times. The agent and its fully-informed counterparts have different needs. Thus, several philosophers have suggested that something is of value to an agent if an ideal version of that agent (fully informed, perfectly rational, etc.) would advise the non-ideal version of the agent to pursue that thing.

This idea of idealizing or extrapolating an agent's preferences¹ goes back at least as far as Sidgwick (1874), who considered the idea that "a man's future good" consists in "what he would now desire... if all the consequences of all the different [actions] open to him were accurately forseen..." Similarly, Rawls (1971) suggested that a person's good is the plan "that would be decided upon as the outcome of careful reflection in which the agent reviewed, in the light of all the relevant facts, what it would be like to carry out these plans..." More recently, in an article about rational agents and moral theory, Harsanyi (1982) defined what an agent's rational wants as “the preferences he would have if he had all the relevant factual information, always reasoned with the greatest possible care, and were in a state of mind most conducive to rational choice.” Then, a few years later, Railton (1986) identified a person's good with "what he would want himself to want... were he to contemplate his present situation from a standpoint fully and vividly informed about himself and his circumstances, and entirely free of cognitive error or lapses of instrumental rationality."

Rosati (1995) calls these theories Ideal Advisor theories of value because they identify one's personal value with what an ideal version of oneself would advise the non-ideal self to value.

Looking not for a metaphysical account of value but for a practical solution to machine ethics (Wallach & Allen 2009; Muehlhauser & Helm 2012), Yudkowsky (2004) described a similar concept which he calls "coherent extrapolated volition" (CEV):

In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

In other words, the CEV of humankind is about the preferences that we would have as a species if our preferences were extrapolated in certain ways. Armed with this concept, Yudkowsky then suggests that we implement CEV as an "initial dynamic" for "Friendly AI." Tarleton (2010) explains that the intent of CEV is that "our volition be extrapolated once and acted on. In particular, the initial extrapolation could generate an object-level goal system we would be willing to endow a superintelligent [machine] with."

CEV theoretically avoids many problems with other approaches to machine ethics (Yudkowsky 2004; Tarleton 2010; Muehlhauser & Helm 2012). However, there are reasons it may not succeed. In this post, we examine one such reason: Resolving CEV at the level of humanity (Global CEV) might require at least partially resolving CEV at the level of individuals (Personal CEV)², but Personal CEV is similar to ideal advisor theories of value,³ and such theories face well-explored difficulties. As such, these difficulties may undermine the possibility of determining the Global CEV of humanity.

Before doing so, however, it's worth noting one key difference between Ideal Advisor theories of value and Personal CEV. Ideal Advisor theories typically are linguistic or metaphysical theories, while the role of Personal CEV is epistemic. Ideal Advisor theorists attempts to define what it is for something to be of value for an agent. Because of this, their accounts needs to give an unambiguous and plausible answer in all cases. On the other hand, Personal CEV's role is an epistemic one: it isn't intended to define what is of value for an agent. Rather, Personal CEV is offered as a technique that can help an AI to come to know, to some reasonable but not necessarily perfect level of accuracy, what is of value for the agent. To put it more precisely, Personal CEV is intended to allow an initial AI to determine what sort of superintelligence to create such that we end up with what Yudkowsky calls a "Nice Place to Live." Given this, certain arguments are likely to threaten Ideal Advisor theories and not to Personal CEV, and vice versa.

With this point in mind, we now consider some objections to ideal advisor theories of value, and examine whether they threaten Personal CEV.

I propose it is altruistic to be replaceable and therefore, those who strive to be altruistic should strive to be replaceable.

As far as I can Google, this does not seem to have been proposed before. LW should be a good place to discuss it. A community interested in rational and ethical behavior, and in how superintelligent machines may decide to replace mankind, should at least bother to refute the following argument.

Replaceability

Replaceability is "the state of being replaceable". It isn't binary. The price of the replacement matters: so a cookie is more replaceable than a big wedding cake. Adequacy of the replacement also makes a difference: a piston for an ancient Rolls Royce is less replaceable than one in a modern car, because it has to be hand-crafted and will be distinguishable. So something is more or less replaceable depending on the price and quality of its replacement.

Replaceability could be thought of as the inverse of the cost of having to replace something. Something that's very replaceable has a low cost of replacement, while something that lacks replaceability has a high (up to unfeasible) cost of replacement. The cost of replacement plays into Total Cost of Ownership, and everything economists know about that applies. It seems pretty obvious that replaceability of possessions is good, much like cheap availability is good.

Some things (historical artifacts, art pieces) are valued highly precisely because of their irreplacability. Although a few things could be said about the resale value of such objects, I'll simplify and contend these valuations are not rational.

The practical example

Anne manages the central database of Beth's company. She's the only one who has access to that database, the skillset required for managing it, and an understanding of how it all works; she has a monopoly to that combination.

This monopoly gives Anne control over her own replacement cost. If she works according to the state of the art, writes extensive and up-to-date documentation, makes proper backups etc she can be very replaceable, because her monopoly will be easily broken. If she refuses to explain what she's doing, creates weird and fragile workarounds and documents the database badly she can reduce her replaceability and defend her monopoly. (A well-obfuscated database can take months for a replacement database manager to handle confidently.)

So Beth may still choose to replace Anne, but Anne can influence how expensive that'll be for Beth. She can at least make sure her replacement needs to be shown the ropes, so she can't be fired on a whim. But she might go further and practically hold the database hostage, which would certainly help her in salary negotiations if she does it right.

This makes it pretty clear how Anne can act altruistically in this situation, and how she can act selfishly. Doesn't it?

The moral argument

To Anne, her replacement cost is an externality and an influence on the length and terms of her employment. To maximize the length of her employment and her salary, her replacement cost would have to be high.

To Beth, Anne's replacement cost is part of the cost of employing her and of course she wants it to be low. This is true for any pair of employer and employee: Anne is unusual only in that she has a great degree of influence on her replacement cost.

Therefore, if Anne documents her database properly etc, this increases her replaceability and constitutes altruistic behavior. Unless she values the positive feeling of doing her employer a favor more highly than she values the money she might make by avoiding replacement, this might even be true altruism.

Unless I suck at Google, replaceability doesn't seem to have been discussed as an aspect of altruism. The two reasons for that I can see are:

• replacing people is painful to think about
• and it seems futile as long as people aren't replaceable in more than very specific functions anyway.

But we don't want or get the choice to kill one person to save the life of five, either, and such practical improbabilities shouldn't stop us from considering our moral decisions. This is especially true in a world where copies, and hence replacements, of people are starting to look possible at least in principle.

Singularity-related hypotheticals

1. In some reasonably-near future, software is getting better at modeling people. We still don't know what makes a process intelligent, but we can feed a couple of videos and a bunch of psychological data points into a people modeler, extrapolate everything else using a standard population and the resulting model can have a conversation that could fool a four-year-old. The technology is already good enough for models of pets. While convincing models of complex personalities are at least another decade away, the tech is starting to become good enough for senile grandmothers.
   
   Obviously no-one wants granny to die. But the kids would like to keep a model of granny, and they'd like to make the model before the Alzheimer's gets any worse, while granny is terrified she'll get no more visits to her retirement home.
   
   What's the ethical thing to do here? Surely the relatives should keep visiting granny. Could granny maybe have a model made, but keep it to herself, for release only through her Last Will and Testament? And wouldn't it be truly awful of her to refuse to do that?
2. Only slightly further into the future, we're still mortal, but cryonics does appear to be working. Unfrozen people need regular medical aid, but the technology is only getting better and anyway, the point is: something we can believe to be them can indeed come back.
   
   Some refuse to wait out these Dark Ages; they get themselves frozen for nonmedical reasons, to fastforward across decades or centuries into a time when the really awesome stuff will be happening, and to get the immortality technologies they hope will be developed by then.
   
   In this scenario, wouldn't fastforwarders be considered selfish, because they impose on their friends the pain of their absence? And wouldn't their friends mind it less if the fastforwarders went to the trouble of having a good model (see above) made first?
3. On some distant future Earth, minds can be uploaded completely. Brains can be modeled and recreated so effectively that people can make living, breathing copies of themselves and experience the inability to tell which instance is the copy and which is the original.
   
   Of course many adherents of soul theories reject this as blasphemous. A couple more sophisticated thinkers worry if this doesn't devalue individuals to the point where superhuman AIs might conclude that as long as copies of everyone are stored on some hard drive orbiting Pluto, nothing of value is lost if every meatbody gets devoured into more hardware. Bottom line is: Effective immortality is available, but some refuse it out of principle.
   
   In this world, wouldn't those who make themselves fully and infinitely replaceable want the same for everyone they love? Wouldn't they consider it a dreadful imposition if a friend or relative refused immortality? After all, wasn't not having to say goodbye anymore kind of the point?

These questions haven't come up in the real world because people have never been replaceable in more than very specific functions. But I hope you'll agree that if and when people become more replaceable, that will be regarded as a good thing, and it will be regarded as virtuous to use these technologies as they become available, because it spares one's friends and family some or all of the cost of replacing oneself.

Replaceability as an altruist virtue

And if replaceability is altruistic in this hypothetical future, as well as in the limited sense of Anne and Beth, that implies replaceability is altruistic now. And even now, there are things we can do to increase our replaceability, i.e. to reduce the cost our bereaved will incur when they have to replace us. We can teach all our (valuable) skills, so others can replace us as providers of these skills. We can not have (relevant) secrets, so others can learn what we know and replace us as sources of that knowledge. We can endeavour to live as long as possible, to postpone the cost. We can sign up for cryonics. There are surely other things each of us could do to increase our replaceability, but I can't think of any an altruist wouldn't consider virtuous.

As an altruist, I conclude that replaceability is a prosocial, unselfish trait, something we'd want our friends to have, in other words: a virtue. I'd go as far as to say that even bothering to set up a good Last Will and Testament is virtuous precisely because it reduces the cost my bereaved will incur when they have to replace me. And although none of us can be truly easily replaceable as of yet, I suggest we honor those who make themselves replaceable, and are proud of whatever replaceability we ourselves attain.

So, how replaceable are you?

Peter Singer, makes a (refreshingly simple) ethical case for anti-aging research, and endorses increased funding.

http://www.project-syndicate.org/commentary/the-ethics-of-anti-aging-by-peter-singer

On which problems should we focus research in medicine and the biological sciences? There is a strong argument for tackling the diseases that kill the most people –diseases like malaria, measles, and diarrhea, which kill millions in developing countries, but very few in the developed world.

Developed countries, however, devote most of their research funds to the diseases from which their citizens suffer, and that seems likely to continue for the foreseeable future. Given that constraint, which medical breakthrough would do the most to improve our lives?

If your first thought is “a cure for cancer” or “a cure for heart disease,” think again. Aubrey de Grey, Chief Science Officer of SENS Foundation and the world’s most prominent advocate of anti-aging research, argues that it makes no sense to spend the vast majority of our medical resources on trying to combat the diseases of aging without tackling aging itself. If we cure one of these diseases, those who would have died from it can expect to succumb to another in a few years. The benefit is therefore modest.

[...]

De Grey has set up SENS Foundation to promote research into anti-aging. By most standards, his fundraising efforts have been successful, for the foundation now has an annual budget of around $4 million. But that is still pitifully small by the standards of medical research foundations. De Grey might be mistaken, but if there is only a small chance that he is right, the huge pay-offs make anti-aging research a better bet than areas of medical research that are currently far better funded.

In Robert Nozick's famous "Utility Monster" thought experiment he proposes the idea of a creature that does not receive diminishing marginal utility from resource consumption, and argues that this poses a problem for utilitarian ethics.  Why?  Utilitarian ethics, while highly egalitarian in real life situations, does not place any intrinsic value on equality.  The reason utilitarian ethics tend to favor equality is that human beings seem to experience diminishing returns when converting resources into utility.  Egalitarianism, according to this framework, is good because sharing resources between people reduces the level of diminishing returns and maximizes the total amount of utility people generate, not because it's actually good for people to have equal levels of utility.

The problem the Utility Monster poses is that, since it does not receive diminishing marginal utility, there is no reason, under traditional utilitarian framework, to share resources between it and the other inhabitants of the world it lives in.  It would be completely justified in killing other people and taking their things for itself, or enslaving them for its own benefit.  This seems counter-intuitive to Nozick, and many other people.

There seem to be two possible reasons for this.  One, of course, is that most people's intuitions are wrong in this particular case.  The reason I am interesting in exploring, however, is the other one, namely that equality is valuable for its own sake, not just as a side effect of diminishing marginal utility.

Now, before I go any further I should clarify what I mean by "equality."  There are many different types of equality, not all of which are compatible with each other.  What I mean is equality of utility, everyone has the same level of satisfied preferences, happiness, and whatever else "utility" constitutes.  This is not the same thing as fiscal equality, as some people may differ in their ability to convert money and resources into utility (people with horrible illnesses, for instance, are worse at doing so than the general population).  It is also important to stress that "lifespan" should be factored in as part of the utility that is to be equalized (i.e. killing someone increases inequality).  Otherwise one could achieve equality of utility by killing all the poor people.

So if equality is valuable for its own sake, how does one factor it into utilitarian calculations?  It seems wrong to replace utility maximization with equality maximization.  That would imply that a world where everyone had 10 utilons and a society where everyone had 100 utilons are morally identical, which seems wrong, to say the least. 

What about making equality lexically prior to utility maximization?  That seems just as bad.  It would imply, among other things, that in a stratified world where some people have far greater levels of utility than others, that it would be morally right to take an action that would harm every single person in the world, as long as it hurt the best off slightly more than the worst off.  That seems insanely wrong.  The Utility Monster thought experiment already argues against making utility maximization lexically prior to equality.

So it seems like the best option would be to have maximizing utility and increasing equality as two separate values.  How then, to trade one off against the other?  If there is some sort of straight, one-to-one value then this doesn't do anything to dismiss the problem of the Utility Monster.  A monster good enough at utility generation could simply produce so much utility that no amount of equality could equal its output.

The best possible solution I can see would be to have utility maximization and equality have diminishing returns relative to each other.  This would mean that in a world with high equality, but low utility, raising utility would be more important, while in a world of low equality and high utility, establishing equality would be more important.

This solution deals with the utility monster fairly effectively.  No matter how much utility the monster can generate, it is always better to share some of its resources with other people.

Now, you might notice that this doesn't eliminate every aspect of the utility monster problem.  As long as the returns generated by utility maximization do not diminish to zero you can always posit an even more talented monster.  And you can then argue that the society created by having that monster enslave the rest of the populace is better than one where a less talented monster shares with the rest of the populace. However, this new society would instantly become better if the new Utility Monster was forced to share its resources with the rest of the population.

This is a huge improvement over the old framework.  Ordinary utility maximizing ethics would not only argue that a world where a Utility Monster enslaved everyone else might be a better world.  They would argue that it was the optimal world, the best possible world given the constraints the inhabitants face.  Under this new ethical framework, however, that is never the case.  The optimal world, under any given level of constraints, is one where a utility monster shares with the rest of the population. 

In other words, under this framework, if you were to ask, "Is it good for a utility monster to enslave the rest of the population?" the answer would always be "No."

Obviously the value of equality has many other aspects to be considered.  For instance is it better described by traditional egalitarianism, or by prioritarianism?  Values are often more complex than they first appear.

It also seems quite possible that there are other facets of value besides maximizing utility and equality of utility.  For instance, total and average utilitarianism might be reconciled by making them two separate values that are both important.  Other potential candidates include prioritarian concerns (if they are not included already), number of worthwhile lives (most people would consider a world full of people with excellent lives better than one inhabited solely by one ecstatic utility monster), consideration of prior-existing people, and perhaps many, many more. As with utility and equality, these values would have diminishing returns relative to each other, and an optimum society would be one where all receive some measure of consideration.

An aside.  This next section is not directly related to the rest of the essay, but develops the idea in a direction I thought was interesting:

It seems to me that the value of equality could be the source of a local disagreement in population ethics.  There are several people (Robin Hanson, most notably) who have argued that it would be highly desirable to create huge amounts of poor people with lives barely worth living, and that this may well be better than having a smaller, wealthier population. Many other people consider this to be a bad idea.

The unspoken assumption in this argument is that multiple lives barely worth living generate more utility than a single very excellent life. At first this seems like an obvious truth, based on the following chain of logic:

1. It is obviously wrong for Person A, who has a life barely worth living, to kill Person B, who also has a life barely worth living, and use B's property to improve their own life.

2. The only reason something is wrong is that it decreases the level of utility.

3. Therefore, killing Person B must decrease the level of utility.

4. Therefore, two lives barely worth living must generate more utility than a single excellent life.

However, if equality is valued for its own sake, then the reason it is wrong to kill Person B might be because of the vast inequality in various aspects of utility (lifespan, for instance) that their death would create between A and B.

This means that a society that has a smaller population living great lives might very well be generating a much larger amount of utility than a larger society whose inhabitants live lives barely worth living.

I'm Anja Heinisch, the new visiting fellow at SI. I've been researching replacing AIXI's reward system with a proper utility function. Here I will describe my AIXI+utility function model, address concerns about restricting the model to bounded or finite utility, and analyze some of the implications of modifiable utility functions, e.g. wireheading and dynamic consistency. Comments, questions and advice (especially about related research and material) will be highly appreciated.

Introduction to AIXI

Marcus Hutter's (2003) universal agent AIXI  addresses the problem of rational action in a (partially) unknown computable universe, given infinite computing power and a halting oracle. The agent interacts with its environment in discrete time cycles, producing an action-perception sequence  with actions (agent outputs)   and perceptions (environment outputs)   chosen from finite sets  and . The perceptions are pairs , where  is the observation part and  denotes a reward. At time k the agent chooses its next action  according to the expectimax principle:

Here M denotes the updated Solomonoff prior summing over all programs  that are consistent with the history  ^[1] and which will, when run on the universal Turing machine T with successive inputs , compute outputs , i.e.

AIXI is a dualistic framework in the sense that the algorithm that constitutes the agent is not part of the environment, since it is not computable. Even considering that any running implementation of AIXI would have to be computable, AIXI accurately simulating AIXI accurately simulating AIXI ad infinitem doesn't really seem feasible. Potential consequences of this separation of mind and matter include difficulties the agent may have predicting the effects of its actions on the world. 

Utility vs rewards

So, why is it a bad idea to work with a reward system? Say the AIXI agent is rewarded whenever a human called Bob pushes a button. Then a sufficiently smart AIXI will figure out that instead of furthering Bob’s goals it can also threaten or deceive Bob into pushing the button, or get another human to replace Bob. On the other hand, if the reward is computed in a little box somewhere and then displayed on a screen, it might still be possible to reprogram the box or find a side channel attack. Intuitively you probably wouldn't even blame the agent for doing that -- people try to game the system all the time. 

You can visualize AIXI's computation as maximizing bars displayed on this screen; the agent is unable to connect the bars to any pattern in the environment, they are just there. It wants them to be as high as possible and it will utilize any means at its disposal. For a more detailed analysis of the problems arising through reinforcement learning, see Dewey (2011).

Is there a way to bind the optimization process to actual patterns in the environment? To design a framework in which the screen informs the agent about the patterns it should optimize for? The answer is, yes, we can just define a utility function

that assigns a value  to every possible future history  and use it to replace the reward system in the agent specification:

When I say "we can just define" I am actually referring to the really hard question of how to recognize and describe the patterns we value in the universe. Contrasted with the necessity to specify rewards in the original AIXI framework, this is a strictly harder problem, because the utility function has to be known ahead of time and the reward system can always be represented in the framework of utility functions by setting

For the same reasons, this is also a strictly safer approach.

Infinite utility

The original AIXI framework must necessarily place upper and lower bound on the rewards that are achievable, because the rewards are part of the perceptions and  is finite. The utility function approach does not have this problem, as the expected utility 

is always finite as long as we stick to a finite set of possible perceptions, even if the utility function is not bounded. Relaxing this constraint and allowing  to be infinite and the utility to be unbounded creates divergence of expected utility (for a proof see de Blanc 2008). This closely corresponds to the question of how to be a consequentialist in an infinite universe, discussed by Bostrom (2011). The underlying problem here is that (using the standard approach to infinities) these expected utilities will become incomparable. One possible solution to this problem could be to use a larger subfield than  of the surreal numbers, my favorite^[2] so far being the Levi-Civita field generated by the infinitesimal :

with the usual power-series addition and multiplication. Levi-Civita numbers can be written and approximated as 

(see Berz 1996), which makes them suitable for representation on a computer using floating point arithmetic. If we allow the range of our utility function to be , we gain the possibility of generalizing the framework to work with an infinite set of possible perceptions, therefore allowing for continuous parameters. We also allow for a much broader set of utility functions, no longer excluding the assignment of infinite (or infinitesimal) utility to a single event. I recently met someone who argued convincingly that his (ideal) utility function assigns infinite negative utility to every time instance that he is not alive, therefore making him prefer life to any finite but huge amount of suffering.

Note that finiteness of  is still needed to guarantee the existence of actions with maximal expected utility, and the finite (but dynamic) horizon  remains a very problematic assumption, as described in Legg (2008).

Modifiable utility functions

Any implementable approximation of AIXI implies a weakening of the underlying dualism. Now the agent's hardware is part of the environment and at least in the case of a powerful agent, it can no longer afford to neglect the effect its actions may have on its source code and data. One question that has been asked is whether AIXI can protect itself from harm. Hibbard (2012) shows that an agent similar to the one described above, equipped with the ability to modify its policy responsible for choosing future actions, would not do so, given that it starts out with the (meta-)policy to always use the optimal policy, and the additional constraint to change only if that leads to a strict improvement. Ring and Orseau (2011) study under which circumstances a universal agent would try to tamper with the sensory information it receives. They introduce the concept of a delusion box, a device that filters and distorts the perception data before it is written into the part of the memory that is read during the calculation of utility. 

A further complication to take into account is the possibility that the part of memory that contains the utility function may get rewritten, either by accident, by deliberate choice (programmers trying to correct a mistake), or in an attempt to wirehead. To analyze this further we will now consider what can happen if the screen flashes different goals in different time cycles. Let 

denote the utility function the agent will have at time k.

Even though we will only analyze instances in which the agent knows at time k, which utility function  it will have at future times  (possibly depending on the actions  before that), we note that for every fixed future history  the agent knows the utility function  that is displayed on the screen because the screen is part of its perception data .

This leads to three different agent models worthy of further investigation:

• Agent 1 will optimize for the goals that are displayed on the screen right now and act as if it would continue to do so in the future. We describe this with the utility function   
• Agent 2 will try to anticipate future changes to its utility function and maximize the utility it experiences at every time cycle as shown on the screen at that time. This is captured by 
• Agent 3 will, at time k, try to maximize the utility it derives in hindsight, displayed on the screen at the time horizon  

Of course arbitrary mixtures of these are possible.

The type of wireheading that is of interest here is captured by the Simpleton Gambit described by Orseau and Ring (2011), a Faustian deal that offers the agent maximal utility in exchange for its willingness to be turned into a Simpleton that always takes the same default action at all future times. We will first consider a simplified version of this scenario: The Simpleton future, where the agent knows for certain that it will be turned into a Simpleton at time k+1, no matter what it does in the remaining time cycle. Assume that for all possible action-perception combinations the utility given by the current utility function is not maximal, i.e.   holds for all . Assume further that the agents actions influence the future outcomes, at least from its current perspective. That is, for all  there exist   with . Let  be the Simpleton utility function, assigning equal but maximal utility  to all possible futures. While Agent 1 will optimize as before, not adapting its behavior to the knowledge that its utility function will change, Agent 3 will be paralyzed, having to rely on whatever method its implementation uses to break ties. Agent 2 on the other hand will try to maximize only the utility .

Now consider the actual Simpleton Gambit: At time k the agent gets to choose between changing, , resulting in  and  (not changing), leading to  for all . We assume that  has no further effects on the environment. As before, Agent 1 will optimize for business as usual, whether or not it chooses to change depends entirely on whether the screen specifically mentions the memory pointer to the utility function or not.

Agent 2 will change if and only if the utility of changing compared to not changing according to what the screen currently says is strictly smaller than the comparative advantage of always having maximal utility in the future. That is,

is strictly less than

This seems quite analogous to humans, who sometimes tend to choose maximal bliss over future optimization power, especially if the optimization opportunities are meager anyhow. Many people do seem to choose their goals so as to maximize the happiness felt by achieving them at least some of the time; this is also advice that I have frequently encountered in self-help literature, e.g. here. Agent 3 will definitely change, as it only evaluates situations using its final utility function.

Comparing the three proposed agents, we notice that Agent 1 is dynamically inconsistent: it will optimize for future opportunities, that it predictably will not take later. Agent 3 on the other hand will wirehead whenever possible (and we can reasonably assume that opportunities to do so will exist in even moderately complex environments). This leaves us with Agent model 2 and I invite everyone to point out its flaws.

[1] Dotted actions/ perceptions, like  denote past events, underlined perceptions  denote random variables to be observed at future times.

[2] Bostrom (2011) proposes using hyperreal numbers, which rely heavily on the axiom of choice for the ultrafilter to be used and I don't see how those could be implemented.

I was surprised to see the high number of moral realists on Less Wrong, so I thought I would bring up a (probably unoriginal) point that occurred to me a while ago.

Let's say that all your thoughts either seem factual or fictional.  Memories seem factual, stories seem fictional.  Dreams seem factual, daydreams seem fictional (though they might seem factual if you're a compulsive fantasizer).  Although the things that seem factual match up reasonably well to the things that actually are factional, this isn't the case axiomatically.  If deviating from this pattern is adaptive, evolution will select for it.  This could result in situations like: the rule that pieces move diagonally in checkers seems fictional, while the rule that you can't kill people seems factual, even though they're both just conventions.  (Yes, the rule that you can't kill people is a very good convention, and it makes sense to have heavy default punishments for breaking it.  But I don't think it's different in kind from the rule that you must move diagonally in checkers.)

I'm not an expert, but it definitely seems as though this could actually be the case.  Humans are fairly conformist social animals, and it seems plausible that evolution would've selected for taking the rules seriously, even if it meant using the fact-processing system for things that were really just conventions.

Another spin on this: We could see philosophy as the discipline of measuring, collating, and making internally consistent our intuitions on various philosophical issues.  Katja Grace has suggested that the measurement of philosophical intuitions may be corrupted by the desire to signal on the part of the philosophy enthusiasts.  Could evolutionary pressure be an additional source of corruption?  Taking this idea even further, what do our intuitions amount to at all aside from a composite of evolved and encultured notions?  If we're talking about a question of fact, one can overcome evolution/enculturation by improving one's model of the world, performing experiments, etc.  (I was encultured to believe in God by my parents.  God didn't drop proverbial bowling balls from the sky when I prayed for them, so I eventually noticed the contradiction in my model and deconverted.  It wasn't trivial--there was a high degree of enculturation to overcome.)  But if the question has no basis in fact, like the question of whether morals are "real", then genes and enculturation will wholly determine your answer to it.  Right?

Yes, you can think about your moral intuitions, weigh them against each other, and make them internally consistent.  But this is kind of like trying to add resolution back in to an extremely pixelated photo--just because it's no longer obviously "wrong" doesn't guarantee that it's "right".  And there's the possibility of path-dependence--the parts of the photo you try to improve initially could have a very significant effect on the final product.  Even if you think you're willing to discard your initial philosophical conclusions, there's still the possibility of accidentally destroying your initial intuitional data or enculturing yourself with your early results.

To avoid this possibility of path-dependence, you could carefully document your initial intuitions, pursue lots of different paths to making them consistent in parallel, and maybe even choose a "best match".  But it's not obvious to me that your initial mix of evolved and encultured values even deserves this preferential treatment.

Currently, I disagree with what seems to be the prevailing view on Less Wrong that achieving a Really Good Consistent Match for our morality is Really Darn Important.  I'm not sure that randomness from evolution and enculturation should be treated differently from random factors in the intuition-squaring process.  It's randomness all the way through either way, right?  The main reason "bad" consistent matches are considered so "bad", I suspect, is that they engender cognitive dissonance (e.g. maybe my current ethics says I should hack Osama Bin Laden to death in his sleep with a knife if I get the chance, but this is an extremely bad match for my evolved/encultured intuitions, so I experience a ton of cognitive dissonance actually doing this).  But cognitive dissonance seems to me like just another aversive experience to factor in to my utility calculations.

Now that you've read this, maybe your intuition has changed and you're a moral anti-realist.  But in what sense has your intuition "improved" or become more accurate?

I really have zero expertise on any of this, so if you have relevant links please share them.  But also, who's to say that matters?  In what sense could philosophers have "better" philosophical intuition?  The only way I can think of for theirs to be "better" is if they've seen a larger part of the landscape of philosophical questions, and are therefore better equipped to build consistent philosophical models (example).

The following is a dialogue intended to illustrate what I think may be a serious logical flaw in some of the conclusions drawn from the famous Mere Addition Paradox. 

EDIT:  To make this clearer, the interpretation of the Mere Addition Paradox this post is intended to criticize is the belief that a world consisting of a large population full of lives barely worth living is the optimal world. That is, I am disagreeing with the idea that the best way for a society to use the resources available to it is to create as many lives barely worth living as possible.  Several commenters have argued that another interpretation of the Mere Addition Paradox is that a sufficiently large population with a lower quality of life will always be better than a smaller population with a higher quality of life, even if such a society is far from optimal.  I agree that my argument does not necessarily refute this interpretation, but think the other interpretation is common enough that it is worth arguing against.

EDIT: On the advice of some of the commenters I have added a shorter summary of my argument in non-dialogue form at the end.  Since it is shorter I do not think it summarizes my argument as completely as the dialogue, but feel free to read it instead if pressed for time.

Bob:  Hi, I'm with R&P cable.  We're selling premium cable packages to interested customers.  We have two packages to start out with that we're sure you love.  Package A+ offers a larger selection of basic cable channels and costs $50.  Package B offers a larger variety of exotic channels for connoisseurs,  it costs $100.  If you buy package A+, however, you'll get a 50% discount on B. 

Alice:  That's very nice, but looking at the channel selection, I just don't think that it will provide me with enough utilons.

Bob: Utilons?  What are those?

Alice: They're the unit I use to measure the utility I get from something.  I'm really good at shopping, so if I spend my money on the things I usually spend it on I usually get 1.5 utilons for every dollar I spend.  Now, looking at your cable channels, I've calculated that I will get 10 utilons from buying Package A+ and 100 utilons from buying Package B.  Obviously the total is 110, significantly less than the 150 utilons I'd get from spending $100 on other things.  It's just not a good deal for me.

Bob:  You think so?  Well it so happens that I've met people like you in the past and have managed to convince them.  Let me tell you about something called the "Mere Cable Channel Addition Paradox."

Alice:  Alright, I've got time, make your case.

Bob:  Imagine that the government is going to give you $50.  Sounds like a good thing, right?

Alice:  It depends on where it gets the $50 from.  What if it defunds a program I think is important?

Bob:  Let's say that it would defund a program that you believe is entirely neutral.  The harms the program causes are exactly outweighed by the benefits it brings, leaving a net utility of zero.

Alice:  I can't think of any program like that, but I'll pretend one exists for the sake of the argument.  Yes, defunding it and giving me $50 would be a good thing.

Bob:  Okay, now imagine the program's beneficiaries put up a stink, and demand the program be re-instituted.  That would be bad for you, right?

Alice:  Sure.  I'd be out $50 that I could convert into 75 utilons.

Bob:  Now imagine that the CEO of R&P Cable Company sleeps with an important senator and arranges a deal.  You get the $50, but you have to spend it on Package A+.  That would be better than not getting the money at all, right?

Alice: Sure.  10 utilons is better than zero.  But getting to spend the $50 however I wanted would be best of all.

Bob:  That's not an option in this thought experiment.  Now, imagine that after you use the money you received to buy Package A+, you find out that the 50% discount for Package B still applies.  You can get it for $50.  Good deal, right?

Alice:  Again, sure.  I'd get 100 utilons for $50. Normally I'd only get 75 utilons.

Bob:  Well, there you have it.  By a mere addition I have demonstrated that a world where you have bought both Package A+ and Package B is better than one where you have neither.  The only difference between the hypothetical world I imagined and the world we live in is that in one you are spending money on cable channels.  A mere addition.  Yet you have admitted that that world is better than this one.  So what are you waiting for?  Sign up for Package A+ and Package B!

And that's not all.  I can keep adding cable packages to get the same result.  The end result of my logic, which I think you'll agree is impeccable, is that you purchase Package Z, a package where you spend all the money other than that you need for bare subsistence on cable television packages.

Alice:  That seems like a pretty repugnant conclusion. 

Bob:  It still follows from the logic.  For every world where you are spending your money on whatever you have calculated generates the most utilons there exists another, better world where you are spending all your money on premium cable channels.

Alice:  I think I found a flaw in your logic.  You didn't perform a "mere addition."  The hypothetical world differs from ours in two ways, not one.  Namely, in this world the government isn't giving me $50.  So your world doesn't just differ from this one in terms of how many cable packages I've bought, it also differs in how much money I have to buy them.

Bob: So can I interest you in a special form of the package?  This one is in the form of a legally binding pledge.  You pledge that if you ever make an extra $50 in the future you will use it to buy Package A+.

Alice:  No.  In the scenario you describe the only reason buying Package A+ has any value is that it is impossible to get utility out of that money any other way.  If I just get $50 for some reason it's more efficient for me to spend it normally.

Bob:  Are you sure?  I've convinced a lot of people with my logic.

Alice:  Like who?

Bob:  Well, there were these two customers named Michael Huemer and Robin Hanson who both accepted my conclusion.  They've both mortgaged their homes and started sending as much money to R&P cable as they can.

Alice:  There must be some others who haven't.

Bob:  Well, there was this guy named Derek Parfit who seemed disturbed by my conclusion, but couldn't refute it.  The best he could do is mutter something about how the best things in his life would gradually be lost if he spent all his money on premium cable.  I'm working on him though, I think I'll be able to bring him around eventually.

Alice:  Funny you should mention Derek Parfit.  It so happens that the flaw in your "Mere Cable Channel Addition Paradox" is exactly the same as the flaw in a famous philosophical argument he made, which he called the "Mere Addition Paradox."

Bob:  Really? Do tell?

Alice:  Parfit posited a population he called "A" which had a moderately large population with large amounts of resources, giving them a very high level of utility per person.  Then he added a second population, which was totally isolated from the other population.  How they were isolated wasn't important, although Parfit suggested maybe they were on separate continents and can't sail across the ocean or something like that.  These people don't have nearly as many resources per person as the other population, so each person's level of utility is lower (their lack of resources is the only reason they have lower utility).  However, their lives are still just barely worth living.  He called the two populations "A+."

Parfit asked if "A+" was a better world than "A."  He thought it was, since the extra people were totally isolated from the original population they weren't hurting anyone over there by existing.  And their lives were worth living.  Follow me so far?

Bob: I guess I can see the point.

Alice: Next Parfit posited a population called "B," which was the same as A+. except that the two populations had merged together.  Maybe they got better at sailing across the ocean, it doesn't really matter how.  The people share their resources.  The result is that everyone in the original population had their utility lowered, while everyone in the second had it raised. 

Parfit asked if population "B" was better than "A+" and argued that it was because it had a greater level of equality and total utility.

Bob: I think I see where this is going.  He's going to keep adding more people, isn't he?

Alice:  Yep.  He kept adding more and more people until he reached population "Z," a vast population where everyone had so few resources that their lives were barely worth living.  This, he argued, was a paradox, because he argued that most people would believe that Z is far worse than A, but he had made a convincing argument that it was better.

Bob:  Are you sure that sharing their resources like that would lower the standard of living for the original population?  Wouldn't there be economies of scale and such that would allow them to provide more utility even with less resources per person?

Alice: Please don't fight the hypothetical.  We're assuming that it would for the sake of the argument.

Now, Parfit argued that this argument led to the "Repugnant Conclusion," the idea that the best sort of world is one with a large population with lives barely worth living.  That confers on people a duty to reproduce as often as possible, even if doing so would lower the quality of their and everyone else's lives.

He claimed that the reason his argument showed this was that he had conducted "mere addition."  The populations in his paradox differed in no way other than their size.  By merely adding more people he had made the world "better," even if the level of utility per person plummetted.  He claimed that "For every population, A, with a high average level of utility there exists another, better population, B, with more people and a lower average level of utility."

Do you see the flaw in Parfit's argument? 

Bob:  No, and that kind of disturbs me.  I have kids, and I agree that creating new people can add utility to the world.  But it seems to me that it's also important to enhance the utility of the people who already exist. 

Alice: That's right.  Normal morality tells us that creating new people with lives worth living and enhancing the utility of people that already exist are both good things to use resources on.  Our common sense tells us that we should spend resources on both those things.  The disturbing thing about the Mere Addition Paradox is that it seems at first glance to indicate that that's not true, that we should only devote resources to creating more people with barely worthwhile lives.  I don't agree with that, of course.

Bob:  Neither do I. It seems to me that having a large number of worthwhile lives and a high average utility are both good things and that we should try to increase them both, not just maximize one.

Alice:  You're right, of course.  But don't say "having a high average utility."  Say "use resources to increase the utility of people who already exist."

Bob:  What's the difference? They're the same thing, aren't they?

Alice:  Not quite.  There are other ways to increase average utility than enhancing the utility of existing people.  You could kill all the depressed people, for instance.  Plus, if there was a world where everyone was tortured 24 hours a day, you could increase average utility by creating some new people who are only tortured 23 hours a day.

Bob:  That's insane!  Who could possibly be that literal-minded?

Alice:  You'd be surprised.  The point is, a better way to phrase it is "use resources to increase the utility of people who already exist," not "increase average utility."  Of course, that still leaves some stuff out, like the fact that it's probably better to increase everyone's utility equally, rather than focus on just one person.  But it doesn't lead to killing depressed people, or creating slightly less tortured people in a Hellworld.

Bob:  Okay, so what I'm trying to say is that resources should be used to create people, and to improve people's lives.  Also equality is good. And that none of these things should completely eclipse the other, they're each too valuable to maximize just one.  So a society that increases all of those values should be considered more efficient at generating value than a society that just maximizes one value.  Now that we're done getting our terminology straight, will you tell me what Parfit's mistake was?

Alice:  Population "A" and population "A+" differ in two ways, not one. Think about it.  Parfit is clear that the extra people in "A+" do not harm the existing people when they are added.  That means they do not use any of the original population's resources.  So how do they manage to live lives worth living?  How are they sustaining themselves?

Bob:  They must have their own resources.  To use Parfit's example of continents separated by an ocean;  each continent must have its own set of resources.

Alice:  Exactly.  So "A+" differs from "A" both in the size of its population, and the amount of resources it has access to.  Parfit was not "merely adding" people to the population.  He was also adding resources.

Bob: Aren't you the one who is fighting the hypothetical now?

Alice:  I'm not fighting the hypothetical.  Fighting the hypothetical consists of challenging the likelihood of the thought experiment happening, or trying to take another option than the ones presented.  What I'm doing is challenging the logical coherence of the hypothetical.  One of Parfit's unspoken premises is that you need some resources to live a life worth living, so by adding more worthwhile lives he's also implicitly adding resources.  If he had just added some extra people to population A without giving them their own continent full of extra resources to live on then "A+" would be worse than "A."

Bob:  So the Mere Addition Paradox doesn't confer on us a positive obligation to have as many children as possible, because the amount of resources we have access to doesn't automatically grow with them.  I get that.  But doesn't it imply that as soon as we get some more resources we have a duty to add some more people whose lives are barely worth living?

Alice: No.  Adding lives barely worth living uses the extra resources more efficiently than leaving Parfit's second continent empty for all eternity.  But, it's not the most efficient way.  Not if you believe that creating new people and enhancing the utility of existing people are both important values. 

Let's take population "A+" again.  Now imagine that instead of having a population of people with lives barely worth living, the second continent is inhabited by a smaller population with the same very high percentage of resources and utility per person as the population of the first continent.  Call it "A++. " Would you say "A++" was better than "A+?"

Bob:  Sure, definitely. 

Alice:  How about a world where the two continents exist, but the second one was never inhabited?  The people of the first continent then discover the second one and use its resources to improve their level of utility.

Bob:  I'm less sure about that one, but I think it might be better than "A+."

Alice:  So what Parfit actually proved was: "For every population, A, with a high average level of utility there exists another, better population, B, with more people, access to more resources and a lower average level of utility."

And I can add my own corollary to that:  "For every population, B, there exists another, better population, C, that has the same access to resources as B, but a smaller population and higher average utility."

Bob: Okay, I get it.  But how does this relate to my cable TV sales pitch?

Alice:  Well, my current situation, where I'm spending my money on normal things is analogous to Parfit's population "A."  High utility, and very efficient conversion of resources into utility, but not as many resources.  We're assuming, of course, that using resources to both create new people and improve the utility of existing people is more morally efficient than doing just one or the other.

The situation where the government gives me $50 to spend on Package A+ is analogous to Parfit's population A+.  I have more resources and more utility.  But the resources aren't being converted as efficiently as they could be. 

The situation where I take the 50% discount and buy Package B is equivalent to Parfit's population B.  It's a better situation than A+, but not the most efficient way to use the money.

The situation where I get the $50 from the government to spend on whatever I want is equivalent to my population C.  A world with more access to resources than A, but more efficient conversion of resources to utility than A+ or B.

Bob: So what would a world where the government kept the money be analogous to?

Alice: A world where Parfit's second continent was never settled and remained uninhabited for all eternity, its resources never used by anyone.

Bob: I get it.  So the Mere Addition Paradox doesn't prove what Parfit thought it did?  We don't have any moral obligation to tile the universe with people whose lives are barely worth living?

Alice:  Nope, we don't.  It's more morally efficient to use a large percentage of our resources to enhance the lives of those who already exist.

Bob:  This sure has been a fun conversation.  Would you like to buy a cable package from me?  We have some great deals.

Alice: NO! 

SUMMARY:

My argument is that Parfit’s Mere Addition Paradox doesn’t prove what it seems to.  The argument behind the Mere Addition Paradox is that you can make the world a better place by the “mere addition” of extra people, even if their lives are barely worth living.  In other words : "For every population, A, with a high average level of utility there exists another, better population, B, with more people and a lower average level of utility." This supposedly leads to the Repugnant Conclusion, the belief that a world full of people whose lives are barely worth living is better than a world with a smaller population where the people lead extremely fulfilled and happy lives. 

Parfit demonstrates this by moving from world A, consisting of a population full of people with lots of resources and high average utility, and moving to world A+.  World A+ has an addition population of people who are isolated from the original population and not even aware of the other’s existence. The extra people live lives just barely worth living.  Parfit argues that A+ is a better world than A because everyone in it has lives worth living, and the additional people aren’t hurting anyone by existing because they are isolated from the original population.

Parfit them moves from World A+ to World B, where the populations are merged and share resources.  This lowers the standard of living for the original people and raises it for the newer people.  Parfit argues that B must be better than A+, because it has higher total utility and equality. He then keeps adding people until he reaches Z, a world where everyones’ lives are barely worth living and the population is vast.  He argues that this is a paradox because most people would agree that Z is not a desirable world compared to A.

I argue that the Mere Addition Paradox is a flawed argument because it does not just add people, it also adds resources.  The fact that the extra people in A+ do not harm the original people of A by existing indicates that their population must have a decent amount of resources to live on, even if it is not as many per person as the population of A.  For this reason what the Mere Addition Paradox proves is not that you can make the world better by adding extra people, but rather that you can make it better by adding extra people and resources to support them.  I use a series of choices about purchasing cable television packages to illustrate this in concrete terms.

I further argue for a theory of population ethics that values both using resources to create lives worth living, and using resources to enhance the utility of already existing people, and considers the best sort of world to be one where neither of these two values totally dominate the other.  By this ethical standard A+ might be better than A because it has more people and resources, even if the average level of utility is lower.  However, a world with the same amount of resources as A+, but a lower population and the same, or higher average utility as A is better than A+.

The main unsatisfying thing about my argument is that while it avoids the Repugnant Conclusion in most cases, it might still lead to it, or something close to it, in situations where creating new people and getting new resources are, as one commenter noted, a “package deal.”   In other words, a situation where it is impossible to obtain new resources without creating some new people whose utility levels are below average.  However, even in this case, my argument holds that the best world of all is one where it would be possible to obtain the resources without creating new people, or creating a smaller amount of people with higher utility.

In other words, the Mere Addition Paradox does not prove that: "For every population, A, with a high average level of utility there exists another, better population, B, with more people and a lower average level of utility." Instead what the Mere Addition Paradox seems to demonstrate is that: "For every population, A, with a high average level of utility there exists another, better population, B, with more people, access to more resources and a lower average level of utility."  Furthermore, my own argument demonstrates that: "For every population, B, there exists another, better population, C, which has the same access to resources as B, but a smaller population and higher average utility."

[Looking for feedback, particularly on links to related posts; I'd like to finish this out as a post on the main, provided there aren't too many wrinkles for it to be salvaged.]

Morality as Fixed Computation, Abstracted Idealized Dynamics, as part of the Metaethics Sequence, discuss ethics as computation.  This is a post primarily a response to these two posts, which discuss computation, and the impossibility of computing the full ethical ramifications of an action.  Note that I treat morality as objective, which means, loosely speaking, that two people who share the same ethical values should arrive, provided neither makes logical errors, at approximately the same ethical system.

On to the subject matter of this post - are Bayesian utilitarian ethics utilitarian?  For you?  For most people?

And, more specifically, is a rational ethics system more rational than a heuristics and culturally based one?

I would argue that the answer is, for most people, "No."

The summary explanation of why: Because cultural ethics are functioning ethics.  They have been tested, and work.  They may not be ideal, but most of the "ideal" ethics systems that have been proposed in the past haven't worked.  In terms of Eliezer's post, cultural ethics are the answers that other people have already agreed upon; they are ethical computations which have already been computed, and while there may be errors, most of the potential errors an ethicist might arrive upon have already been weeded out.

The longer explanation of why:

First and foremost, rationality, which I will use from here instead of the word "computation," is -expensive-.  "A witch did it", or the equivalent "Magic!", while not in fact conceptually simple, is in fact logically simple; the complexity is encoded in the concept, not the logic.  The rational explanation for, say, static electricity, requires far more information about the universe, which for an individual who aspires to be a farmer because he likes growing things, may never be useful, and whose internalization may never pay for itself.  It can be fully consistent with a rational attitude to accept irrational explanations, when you have no reasonable expectation that the rational explanation will provide any kind of benefit, or more exactly when the cost of the rational explanation exceeds its expected benefit.

Or, to phrase it another way, it's not always rational to be rational.

Terminal Values versus Instrumental Values discusses some of the computational expenses involved in ethics.  It's a nontrivial problem.

Rationality is a -means-, not an ends.  A "rational ethics system" is merely an ethical system based on logic, on reason.  But if you don't have a rational reason to adopt a rational ethics system, you're failing before you begin; logic is a formalized process, but it's still just a process.  The reason for adopting a rational ethics system is the starting point, the beginning, of that process.  If you don't have a beginning, what do you have?  An ends?  That's not rationality, that's rationalization.

So the very first step in adopting a rational ethics system is determining -why- you want to adopt a rational ethics system.  "I want to be more rational" is irrational.

"I want to know the truth" is a better reason for wanting to be rational.

But the question in turn must, of course, be "Why?"

"Truth has inherent value" isn't an answer, because value isn't inherent, and certainly not to truth.  There is a blue pillow in a cardboard box to my left.  This is a true statement.  You have truth.  Are you more valuable now?  Has this truth enriched your life?  There are some circumstances in which this information might be useful to you, but you aren't in those circumstances, nor in any feasible universe will you be.  It doesn't matter if I lied about the blue pillow.  If truth has inherent value, then every true statement must, in turn, inherit that inherent value.  Not all truth matters.

A rational ethics system must have its axioms.  "Rationality," I hope I have established, is not a useful axiom, nor is "Truth."  It is the values that your ethics system seeks to maximize which are its most important axioms.

The truths that matter are the truths which directly relate to your moral values, to your ethical axioms.  A rational ethics system is a means of maximizing those values - nothing more.

If you have a relatively simple set of axioms, a rational ethics system is relatively simple, if still potentially expensive to compute.  Strict Randian Objectivism, for example, attempts to use human life as its sole primary axiom, which makes it a relatively simple ethical system.  (I'm a less strict Objectivist, and use a different axiom, personal happiness, but this rarely leads to conflict with Randian Objectivism, which uses it as a secondary axiom.)

If, on the other hand, you, like most people, have a wide variety of personal values which you are attempting to maximize, attempting to assess each action on its ethical merits becomes computationally prohibitive.

Which is where heuristics, and inherited ethics, start to become pretty attractive, particularly when you share (and most people do, to more extent than they don't) your culture's ethical values.

If you share at least some of your culture's ethical values, normative ethics can provide immense value to you, by eliminating most of the work necessary in evaluating ethical scenarios.  You don't need to start from the bottom up, and prove to yourself that murder is wrong.  You don't need to weigh the pros and cons of alcoholism.  You don't need to prove that charity is a worthwhile thing to engage in.

"We all engage in ethics, though; it's not like a farmer with static electricity, don't we have a responsibility to understand ethics?"

My flippant response to this question is, should every driver know how to rebuild their car's transmission?

You don't need to be a rationalist in order to reevaluate your ethics.  An expert can rebuild your transmission - an expert can also pose arguments to change your mind.  This has, indeed, happened before on mass scales; racism is no longer broadly acceptable in our society.  It took too long, yes, -but-, a long-established ethics system, being well-tested, should require extraordinary efforts to change.  If it were easily mutable, it would lose much of its value, for it would largely be composed of poorly-tested ideas.

All of which is not to say that rational ethics are inherently irrational - only that one should have a rational reason for engaging in them to begin with.  If you find that societal norms frequently conflict with your own ethical values, that is a good reason to engage in rational ethics.  But if you don't, perhaps you shouldn't.  And if you do, you should be cautious of pushing a rational ethics system on somebody for whom existing ethical systems do well, if your goal is to improve their well-being.

Robin Hanson has done a great job of describing the future world and economy, under the assumption that easily copied "uploads" (whole brain emulations), and the standard laws of economics continue to apply. To oversimplify the conclusion:

• There will be great and rapidly increasing wealth. On the other hand, the uploads will be in Darwinian-like competition with each other and with copies, which will drive their wages down to subsistence levels: whatever is required to run their hardware and keep them working, and nothing more.

The competition will not so much be driven by variation, but by selection: uploads with the required characteristics can be copied again and again, undercutting and literally crowding out any uploads wanting higher wages.

Megadeaths

Some have focused on the possibly troubling aspects voluntary or semi-voluntary death: some uploads would be willing to make copies of themselves for specific tasks, which would then be deleted or killed at the end of the process. This can pose problems, especially if the copy changes its mind about deletion. But much more troubling is the mass death among uploads that always wanted to live.

What the selection process will favour is agents that want to live (if they didn't, they'd die out) and willing to work for an expectation of subsistence level wages. But now add a little risk to the process: not all jobs pay exactly the expected amount, sometimes they pay slightly higher, sometimes they pay slightly lower. That means that half of all jobs will result in a life-loving upload dying (charging extra to pay for insurance will squeeze that upload out of the market). Iterating the process means that the vast majority of the uploads will end up being killed - if not initially, then at some point later. The picture changes somewhat if you consider "super-organisms" of uploads and their copies, but then the issue simply shifts to wage competition between the super-organisms.

The only way this can be considered acceptable is if the killing of a (potentially unique) agent that doesn't want to die, is exactly compensated by the copying of another already existent agent. I don't find myself in the camp arguing that that would be a morally neutral or positive action.

Pain and unhappiness

The Economist has a new article on ethical dilemmas faced by machine designers.

Evidently:

1. In the event of an immoral decision by a machine, neural networks make it too hard to know who is at fault--the programmer, the operator, the manufacturer, or the designer. Thus, neural networks might be a bad idea.

2. Robots' ethical systems ought to resonate with "most people."

3. Proper robot consciences are more likely to arise given greater collaboration among engineers, ethicists, policymakers, and lawyers. Key quotation:

Both ethicists and engineers stand to benefit from working together: ethicists may gain a greater understanding of their field by trying to teach ethics to machines, and engineers need to reassure society that they are not taking any ethical short-cuts.

The second clause of the above sentence is quite similar to something Yudkowsky wrote, perhaps more than once, about the value of approaching ethics from an AI standpoint. I do not recall where he wrote it, nor did my search turn up the appropriate post.

It’s always good news when someone else develops an idea independently from you.  It's a sign you might be onto something.  Which is why I was excited to discover that Alan Carter, Professor Emeritus of the University of Glasgow’s Department of Philosophy, has developed the concept of Complexity of Value independent of Less Wrong. 

As far as I can tell Less Wrong does not know of Carter, the only references to his existence I could find on LW and OB were written by me.  Whether Carter knows of LW or OB is harder to tell, but the only possible link I could find online was that he has criticized the views of Michael Huemer, who knows Bryan Caplan, who knows Robin Hanson. This makes it all the more interesting that Carter has developed views on value and morality very similar to ones commonly espoused on Less Wrong.

The Complexity of Value is one of the more important concepts in Less Wrong.  It has been elaborated on its wiki page, as well as some classic posts by Eliezer.  Carter has developed the same concept in numerous papers, although he usually refers to it as “a plurality of values” or “multidimensional axiology of value.”  I will focus the discussion on working papers Carter has on the University of Glasgow’s website, as they can be linked to directly without having to deal with a pay wall.  In particular I will focus on his paper "A Plurality of Values."

Carter begins the paper by arguing:

Wouldn’t it be nice if we were to discover that the physical universe was reducible to only one kind of fundamental entity? ... Wouldn’t it be nice, too, if we were to discover that the moral universe was reducible to only one kind of valuable entity—or one core value, for short? And wouldn’t it be nice if we discovered that all moral injunctions could be derived from one simple principle concerning the one core value, with the simplest and most natural thought being that we should maximize it? There would be an elegance, simplicity and tremendous justificatory power displayed by the normative theory that incorporated the one simple principle. The answers to all moral questions would, in theory at least, be both determinate and determinable. It is hardly surprising, therefore, that many moral philosophers should prefer to identify, and have thus sought, the one simple principle that would, hopefully, ground morality.

And it is hardly surprising that many moral philosophers, in seeking the one simple principle, should have presumed, explicitly or tacitly, that morality must ultimately be grounded upon the maximization of a solitary core value, such as quantity of happiness or equality, say. Now, the assumption—what I shall call the presumption of value-monism—that here is to be identified a single core axiological value that will ultimately ground all of our correct moral decisions has played a critical role in the development of ethical theory, for it clearly affects our responses to certain thought-experiments, and, in particular, our responses concerning how our normative theories should be revised or concerning which ones ought to be rejected.

Most members of this community will immediately recognize the similarities between these paragraphs and Eliezer’s essay “Fake Utility Functions.”  The presumption of value monism sounds quite similar to Eliezer’s description of “someone who has discovered the One Great Moral Principle, of which all other values are a mere derivative consequence.”  Carter's opinion of such people is quite similar to Eliezer's. 

While Eliezer discovered the existence of the Complexity of Value by working on Friendly AI, Carter discovered it by studying some of the thornier problems in ethics, such as the Mere Addition Paradox and what Carter calls the Problem of the Ecstatic Psychopath.  Many Less Wrong readers will be familiar with these problems; they have been discussed numerous times in the community.

For those who aren’t, in brief the Mere Addition Paradox states that if one sets maximizing total wellbeing as the standard of value then one is led to what is commonly called the Repugnant Conclusion, the belief that a huge population of people with lives barely worth living is better than a somewhat smaller population of people with extremely worthwhile lives.  The Problem of the Ecstatic Psychopath is the inverse of this, it states that, if one takes average levels of well-being as the standard of value, that a population of one immortal ecstatic psychopath with a nonsentient machine to care for all their needs is better than a population of trillions of very happy and satisfied, but not ecstatic people.

Carter describes both of these problems in his paper and draws an insightful conclusion:

In short, surely the most plausible reason for the counter-intuitive nature of any mooted moral requirement to bring about, directly or indirectly, the world of the ecstatic psychopath is that either a large total quantity of happiness or a large number of worthwhile lives is of value; and surely the most plausible reason for the counter-intuitive nature of any mooted injunction to bring about, directly or indirectly, the world of the Repugnant Conclusion is that a high level of average happiness is also of value.

How is it that we fail to notice something so obvious? I submit: because we are inclined to dismiss summarily any value that fails to satisfy our desire for the one core value—in other words, because of the presumption of value-monism.

Once Carter has established the faults of value monism he introduces value pluralism to replace it.¹  He introduces two values to start with, “number of worthwhile lives” and “the level of average happiness,” which both contribute to “overall value.”  However,  their contributions have diminishing returns,² so a large population with low average happiness and a tiny population with extremely high average happiness are both  worse than a moderately sized population with moderately high average happiness. 

This is a fairly unique use of the idea of the complexity of value, as far as I know.  I’ve read a great deal of Less Wrong’s discussion of the Mere Addition Paradox, and most attempts to resolve it have consisted of either trying to reformulate Average Utilitarianism so that it does not lead to the Problem of the Ecstatic Psychopath, or redefining what "a life barely worth living" means upwards so that it is much less horrible than one would initially think.  The idea of agreeing that increasing total wellbeing is important, but not the be all and end all of morality, did not seem to come up, although if it did and I missed it I'd be very happy if someone posted a link to that thread.

Carter’s resolution of the Mere Addition Paradox makes a great deal of sense, as it manages to avoid every single repugnant and counterintuitive conclusion that Total and Average Utilitarianism draw by themselves while still being completely logically consistent.  In fact, I think that most people who reject the Repugnant Conclusion will realize that this was their True Rejection all along.  I am tempted to say that Carter has discovered Theory X, the hypothetical theory of population ethics Derek Parfit believed could accurately describe the ethics of creating more people without implying any horrifying conclusions.

Carter does not stop there, however, he then moves to the problem of what he calls “pleasure wizards” (many readers may be more familiar with the term “utility monster”).  The pleasure wizard can convert resources into utility much more efficiently than a normal person, and hence it can be argued that it deserves more resources.  Carter points out that:

…such pleasure-wizards, to put it bluntly, do not exist... But their opposites do. And the opposites of pleasure-wizards—namely, those who are unusually inefficient at converting resources into happiness—suffice to ruin the utilitarian’s egalitarian pretensions. Consider, for example, those who suffer from, what are currently, incurable diseases. … an increase in their happiness would require that a huge proportion of society’s resources be diverted towards finding a cure for their rare condition. Any attempt at a genuine equality of happiness would drag everyone down to the level of these unfortunates. Thus, the total amount of happiness is maximized by diverting resources away from those who are unusually inefficient at converting resources into happiness. In other words, if the goal is, solely, to maximize the total amount of happiness, then giving anything at all to such people and spending anything on cures for their illnesses is a waste of valuable resources. Hence, given the actual existence of such unfortunates, the maximization of happiness requires a considerable inequality in its distribution.

Carter argues that, while most people don’t think all of society’s resources should be diverted to help the very ill, the idea that they should not be helped at all also seems wrong.  He also points out that to a true utilitarian the nonexistence of pleasure wizards should be a tragedy:

So, the consistent utilitarian should greatly regret the non-existence of pleasure-wizards; and the utilitarian should do so even when the existence of extreme pleasure-wizards would morally require everyone else to be no more than barely happy.

Yet, this is not how utilitarians behave, he argues, rather:

As I have yet to meet a utilitarian, and certainly not a monistic one, who admits to thinking that the world would be a better place if it contained an extreme pleasure-wizard living alongside a very large population all at that level of happiness where their lives were just barely worth living…But if they do not  bemoan the lack of pleasure-wizards, then they must surely value equality directly, even if they hide that fact from themselves. And this suggests that the smile of contentment on the faces of utilitarians after they have deployed diminishing marginal utility in an attempt to show that their normative theory is not incompatible with egalitarianism has more to do with their valuing of equality than they are prepared to admit.

Carter resolves the problem of "pleasure wizard" by suggesting equality as an end in itself as a third contributing value towards overall value.  Pleasure wizards should not get all the resources because equality is valuable for its own sake, not just because of diminishing marginal utility.  As with average happiness and total worthwhile lives, equality is balanced against other values, rather than dominating them.   It may often be ethical for a society to sacrifice some amount of equality to increase the total and average wellbeing. 

Carter then briefly states that, though he only discusses three in this paper, there are many other dimensions of value that could be added.  It might even be possible to add some form of deontological rules or virtue ethics to the complexity of value, although  they would be traded off against consequentialist considerations.  He concludes the paper by reiterating that:

Thus, in avoiding the Repugnant Conclusion, the Problem of the Ecstatic Psychopath and the problems posed by pleasure-wizards, as well as the problems posed by any unmitigated demand to level down, we appear to have identified an axiology that is far more consistent with our considered moral judgments than any entailing these counter-intuitive implications.

Carter has numerous other papers discussing the concept in more detail, but “A Plurality of Values” is the most thorough.  Other good ones include “How to solve two addition paradoxes and avoid the Repugnant Conclusion,” which more directly engages the Mere Addition Paradox and some of its defenders like Michael Huemer; "Scrooge and the Pleasure Witch," which discusses pleasure wizards and equality in more detail; and “A pre-emptive response to some possible objections to a multidimensional axiology with variable contributory values,” which is exactly what it says on the tin.

On closer inspection it was not hard to see why Carter had developed theories so close to those of Eliezer and other members of Less Wrong and SIAI communities.   In many ways their two tasks are similar. Eliezer and the SIAI are trying to devise a theory of general ethics that cannot be twisted into something horrible by a rules-lawyering Unfriendly AI, while Carter is trying to devise a theory of population ethics that cannot be twisted into something horrible by rules-lawyering humans.  The worlds of the Repugnant Conclusion and the Ecstatic Psychopath are just the sort of places a poorly programmed AI with artificially simple values would create.

I was very pleased to see an important Less Wrong concept had a defender in mainstream academia.  I was also pleased to see that Carter had not just been content to develop the concept of the Complexity of Value.    He was also able to employ in the concept in new way, successfully resolving one of the major quandaries of modern philosophy.

Footnotes

¹I do not mean to imply Carter developed this theory out of thin air of course. Value pluralism has had many prominent advocates over the years, such as Isaiah Berlin and Judith Jarvis Thomson.

²Theodore Sider proposed a theory called "geometrism" in 1991 that also focused on diminishing returns, but geometrism is still a monist theory, it had geometric diminishing returns for the people in the scenario, rather than the values creating the people was trying to fulfill.

Edited - To remove a reference to Aumann's Agreement Theorem that the commenters convinced me was unnecessary and inaccurate.

This was demonstrated, in a certain limited way, in Peterson (2009). See also Lowry & Peterson (2011).

The Peterson result provides an "asymmetry argument" in favor of consequentialism:

Consequentialists can account for phenomena that are usually thought of in nonconsequentialist terms, such as rights, duties, and virtues, whereas the opposite is false of nonconsequentialist theories. Rights, duty or virtue-based theories cannot account for the fundamental moral importance of consequences. Because of this asymmetry, it seems it would be preferable to become a consequentialist – indeed, it would be virtually impossible not to be a consequentialist.

Another argument in favor of consequentialism has to do with the causes of different types of moral judgments: see Are Deontological Moral Judgments Rationalizations?

Update: see Carl's criticism.

I recently read (in Dawkins' The Ancestor's Tale) about the Alu sequence, and went on to read about transposons generally.  Having as I do a rather broad definition of life, I concluded that Alu (and others like it) are lifeforms in their own right, although parasitic ones.  I found the potential ethical implications somewhat staggering, especially given the need to shut up and multiply those implications by the rather large number of transposon instances in a typical multicellular organism.

I have written out my thoughts on the subject, at http://jttlov.no-ip.org/writings/alulife.htm.  I don't claim to have a well-worked out position, just a series of ideas and questions I feel to be worthy of discussion.

ETA: I have started editing the article based on the discussion below.  For reference with the existing discussion, I have preserved a copy of the original article as well, linked from the current version.

Article here;

http://www.ox.ac.uk/media/science_blog/brainboosting.html

Recent research in Oxford and elsewhere has shown that one type of brain stimulation in particular, called transcranial direct current stimulation or TDCS, can be used to improve language and maths abilities, memory, problem solving, attention, even movement.

Critically, this is not just helping to restore function in those with impaired abilities. TDCS can be used to enhance healthy people’s mental capacities. Indeed, most of the research so far has been carried out in healthy adults.

The article goes on to discuss the ethics of the technique.

Coherent extrapolated volition (CEV) asks what humans would want, if they knew more - if their values reached reflective equilibrium.  (I don't want to deal with the problems of whether there are "human values" today; for the moment I'll consider the more-plausible idea that a single human who lived forever could get smarter and closer to reflective equilibrium over time.)

This is appealing because it seems compatible with moral progress (see e.g., Muehlhauser & Helm, "The singularity and machine ethics", in press).  Morality has been getting better over time, right?  And that's because we're getting smarter, and closer to reflective equilibrium as we revise our values in light of our increased understanding, right?

This view makes three claims:

1. Morality has improved over time.
2. Morality has improved as a result of reflection.
3. This improvement brings us closer to equilibrium over time.

There can be no evidence for the first claim, and the evidence is against the second two claims.

From David Deutsch's The Beginning of Infinity:

Take a powerful computer and set each bit randomly to 0 or 1 using a quantum randomizer. (That means that 0 and 1 occur in histories of equal measure.) At that point all possible contents of the computer’s memory exist in the multiverse. So there are necessarily histories present in which the computer contains an AI program – indeed, all possible AI programs in all possible states, up to the size that the computer’s memory can hold. Some of them are fairly accurate representations of you, living in a virtual-reality environment crudely resembling your actual environment. (Present-day computers do not have enough memory to simulate a realistic environment accurately, but, as I said in Chapter 7, I am sure that they have more than enough to simulate a person.) There are also people in every possible state of suffering. So my question is: is it wrong to switch the computer on, setting it executing all those programs simultaneously in different histories? Is it, in fact, the worst crime ever committed? Or is it merely inadvisable, because the combined measure of all the histories containing suffering is very tiny? Or is it innocent and trivial?

I'm not so sure we have the computing power to "simulate a person," but suppose we did. (Perhaps we will soon.) How would you respond to this worry?

The October 2011 Scientific American has an editorial from its board of editors called "Ban chimp testing", that says:  "In our view, the time has come to end biomedical experimentation on chimpanzees... Chimps should be used only in studies of major diseases and only when there is no other option."  Much of the knowledge described in Luke's recent post on the cognitive science of rationality would have been impossible to acquire under such a ban.

I encourage you to write to Scientific American in favor of chimp testing.  Some points that I plan to make:

• The editors obliquely criticized the NIH to tell the Institute of Medicine to omit ethical considerations from their study of whether chimps are "truly necessary" for biomedical and behavioral research.  But the team tasked with gathering evidence about the necessity of chimps for research shouldn't be making ethical judgements.  They're gathering the data for someone else to make ethical judgements.
• Saying chimps should be used "only when there is no other option" is the same as saying chimps should never be used.  There are always other options.
• This position might be morally defensible if humans were allowed to subject themselves for testing.  The knowledge to be gained from experiment is surely worth the harm to the subject if the subject chooses to undergo the experiment.  Humans are often willing to be test subjects, but aren't allowed to be because of restrictions on human testing.  Banning chimp testing should thus be done only in conjunction with allowing human testing.

I also encourage you to adopt a tone of moral outrage.  Rather than taking the usual apologetic "we're so sorry, but we have to do this awful things in the name of science" tone, get indignant at the editors who intend to harm uncountable numbers of innocent people.  For advanced writers, get indignant not just about harm, but about lost potential, pointing out the ways that our knowledge about how brains work can make our lives better, not just save us from disease.

You can comment on this here, but comments are AFAIK not printed in later issues as letters to the editor.  Actual letters, or at least email, probably have more impact.  You can't submit a letter to the editor through the website, because letters are magically different from things submitted on a website.

ADDED:  Many people responded by claiming that banning chimp experimentation occupies some moral high ground.  That is logically impossible.

To behave morally, you have to do two things:

1. Figure out, inherit, or otherwise acquire a set of moral goals are - let's say, for example, to maximize the sum over all individuals i of all species s of w_s*[pleasure(s,i)-pain(s,i)].

2. Act in a way directed by those moral goals.

If you really cared about the suffering of sentient beings, you would also care about the suffering of humans, and you would realize that there's a tradeoff between the suffering of those experimented on, and of those who benefit, which is different for every experiment.  That's what a moral decision is—deciding how to make a tradeoff of help and harm. People who call for a ban on chimp testing are really demanding we forbid (other) people from making moral judgements and taking moral actions.  There are a wide range of laws and positions that could be argued to be moral.  But just saying "We are incapable of making moral decisions, so we will ban moral decision-making" is not one of them.

Recently I summarized Joshua Greene's attempt to 'explain away' deontological ethics by revealing the cognitive algorithms that generate deontological judgments and showing that the causes of our deontological judgments are inconsistent with normative principles we would endorse.

Mark Alfano has recently done the same thing with virtue ethics (which generally requires a fairly robust theory of character trait possession) in his March 2011 article on the topic: 

I discuss the attribution errors, which are peculiar to our folk intuitions about traits. Next, I turn to the input heuristics and biases, which — though they apply more broadly than just to reasoning about traits — entail further errors in our judgments about trait-possession. After that, I discuss the processing heuristics and biases, which again apply more broadly than the attribution errors but are nevertheless relevant to intuitions about traits... I explain what the biases are, cite the relevant authorities, and draw inferences from them in order to show their relevance to the dialectic about virtue ethics. At the end of the article, I evaluate knowledge-claims about virtues in light of these attribution biases, input heuristics and biases, and processing heuristics and biases. Every widely accepted theory of knowledge must reject such knowledge-claims when they are based merely on folk intuitions.

An overview of the 'situationist' attack on character trait possession can be found in Doris' book Lack of Character.

I'm starting the process of looking for a house to buy.  The first thing every real estate agent says you need to do is to sign an exclusive contract with a real estate agent before they take you to look at houses.

I spoke to some co-workers, and none of them signed the contracts.  I didn't understand:  How can you avoid signing a contract, when the real estate agent, whom you must work with for months, will begin every meeting by telling you that the first thing to do is to sign the contract?

My boss told me that she distracted her agent.  Whenever he brought the subject up, she questioned him about details, which led on to other details, until they were in the car and driving to a showing and talking about something else completely.

This would be a useful skill.  And I can't imagine myself pulling it off.  Something in my gut would twist, and I would choke on my own words, if I tried to use conversation not to communicate information, but to entrap someone into doing something they didn't want to do by ensuring that they would have to violate social conventions to get out of it.

(I asked my boss if she'd ever done that to me.  She smiled very sweetly and said, "Never!")

And even if I could get over the choking, stuttering, and turning red, I don't think I could keep the game up for an entire hour.  I'm inclined to do search, not dynamic control optimization - to play chess (okay, Freecell), not to juggle or do magic tricks.

Somebody who did have the power to do that would be able to do awesome Jedi mind tricks.  (Like, say, pick up women.  Is it a coincidence that some pickup artists are also magicians?)

Are you like me in that way?  Are most of us left-brained Spocks who can't even try to lie or manipulate people?  I've met a lot of you, and I think the answer is "yes", but I really want to see your answers.  If so, is it because you choose to be that way, or because you have no choice?  What is this personality trait that we don't even have a name for, why is rationality so highly-correlated with it, and what else correlates with it?

If you think we're being rational to be so rational, say that, too.

At Wolf Park in Indiana, the biologists, who probably have at least a bit of the nerdy rationalist about them, have developed a technique for dealing with wolves when they're in the enclosure with them.  The wolves interact using dominance displays.  When humans go into the enclosure, and the wolves realize these same people keep coming back (they ignore visitors), the wolves want to establish the places of these humans in the dominance ladder (pedantic note: ladder, not hierarchy).  So they repeatedly try to engage the humans in dominance contests.

The thing about a dominance contest is that quitting equals losing.  You can't opt out of one once it's started (unless you have super Jedi mind skills).  Old-school wolf-handling was to become dominant; the problem with that is that, if you ever go into the wolf enclosure on a day when you have a cold, or are depressed, or just not focused on the task at hand, the nice beta wolf you have regarded as your friend for years may leap on your throat and (at best) throw you to the ground.  Even if you survive, you would be ill-advised to ever go back into the wolf pen; an alpha, once overthrown, moves (strangely) to the very bottom of the dominance ladder, and is fair game for every wolf in the pack.  The no-teaming-up rule which seems to apply to wolf dominance fights (doesn't to primates or felines, BTW) no longer applies.

It's better to leave the question of dominance unsettled.  Just like in the schoolyard, a person with an unknown rank has more leeway than one who is ranked; yet isn't constantly watched for signs of weakness the way an alpha is.  (An alpha wolf at Wolf Park once had a lower spinal injury.  He must have been in agony, but stayed where he was without whining or moving for 2 days before the humans rescued him - not because he couldn't move, but (we think) to keep the other wolves from noticing anything funny about his movements.)  Also, having the alphas be humans disrupts the pack dynamics that the biologists are studying.  So the biologists have developed a strategy of distracting the wolves before they can bring up the question of dominance.  One person does whatever it is they need to go into the enclosure to do; while the rest of them use toys, food, head-skritching, and tag-team techniques to keep any one wolf from focusing on any one human for long.

This is a use of the same technique that my boss used, that is practical and ethically laudable.  Could you do that?  If so, what makes it different?

I still don't think I could do it.  The cognitive load of trying to observe wolves and continually come up with novel distractions would be too great.  I don't think I could do it on the fly - at least, not well enough to ask people to bet their lives on it.  My mind operates with a long clock cycle.  I am CISC, not RISC.  Is that what makes nerds so famously poor at social interaction - that our minds are GOFAI, not cybernetics?

Introduction:

Today there is an almost universal prejudice against individuals with a certain sexual orientation. I am not talking about common homophobia; the prejudice I would like to bring to your attention is so rarely considered a prejudice that it has no particular name. Though the following words will most likely be met with harsh criticism, the prejudice referenced above is the prejudice that almost all of us have against pedophiles. At first thought, it may seem that having a phobia of pedophiles is no more a prejudice for a mother, than having a fear of lions is a prejudice for a mother chimpanzee, but I hope at least to show that the issue is not so clear.

This text does not at any point argue that pedophiles are regular people like you and I, they may well not be. If the hypothesis to be presented is true, however, it follows that the trauma children experience when molested would not happen if we didn't hold the moral judgments towards pedophiles that we do. If this is true then the best thing for us to do as a species for our children is, paradoxically, to stop making the moral judgements we make towards pedophiles. Of course, intuition would have us believe that we hold those moral judgements towards pedophiles precisely because of how traumatic a molestation is for children; this is an attempt to show that that causal interaction goes both ways and forms a loop.

This isn't a defense of pedophilia, nor is it a suggestion that we should stop morally judging pedophiles as a culture, it's an analysis of how circularity can enter the domain of social morality undetected and spread rapidly. We will take a memetic approach to figuring this out, and always ask "how it is useful for the meme to have such and such property?" rather than "how is it useful for us to have a belief with such and such property?".

I will apologize here and now for the graphic nature of this text's subject. But know that part of what I claim is that the reason the following considerations are so rarely even heard is precisely because of their graphic nature. Nowhere in this text is there an argument that can even be loosely interpreted as a defense of individual acts of pedophilia, but the reader may well conclude that in the end, less children would have been seriously hurt if we had refrained from involving our moral attitudes in our dealings with pedophiles.

Inherently Traumatic?:

Let's ask a simple question: "would a feral child be traumatized if molested at a young age?" Notice there was no mention of sodomy in that question. Sodomy is clearly as traumatic to a child as any intense pain caused by another would be. But what about molestation? How can an infant tell the difference between being cleaned and being molested? These two actions could be made to appear behaviorally identical to the child. How does the brain know to get traumatized from one and not from the other? Clearly, children are more frequently traumatized by molestation than by being cleaned. They must somehow make the distinction, either during the act, soon after the event, or retroactively upon remembering the event in adulthood. 

In any case, that distinction must either be learned or inherited. Though we are genetically designed to avoid certain stimuli, e.g., fire, sharp things, bitter chemicals, etc. it is unlikely that getting your genitals touched is one of those stimuli. There might be genes which give you a predisposition to being traumatized when molested as a child, but it is unlikely that we have a sense built into our bodies that distinguishes between acceptable and unacceptable genital touching before puberty. Again, any molestation that causes pain does not apply, we are considering only those cases of molestation which don't cause any physical pain.

If we somehow conclude that any given human does indeed react in a neurologically distinct way when touched on the genitals before puberty by an adult that isn't one of that human's parents, then certainly that sort of molestation would be out of the question. But at the risk of being far to graphic, the fact is that an infant or even a very young child would be largely incapable of distinguishing between grabbing a finger and grabbing an adult male genital. There is clearly nothing inherently evil about the foreskin of a male compared to the skin on his finger. The only difference is the adults intention, which children, or at least infants, are largely insensitive to. What then is the justification for not allowing pedophiles to come to our houses and have our infants reach out and grab their genitals as our infant's instincts would have them do?

It could be argued that children might be traumatized simply by being forced to do something that they do not want to do, and that is certainly likely. But does that mean that we should allow our children to be involved in sexual acts with adults if they are consenting? If we were to argue that children cannot consent, then we would have to ask "can they be non-consenting?" What we generally mean by saying that "children cannot consent." is that they can't consent responsibly because they lack the information to do so. This is granted, but they can simply consent. Children can be made to be the main actors in cases of molestation and even consensual sex. Again, at the risk of being far to graphic: it is not uncommon for one child to molest another, nor is it uncommon for young friends of the same gender to naively engage in games of a sexual nature. Even in the case of molestation from an adult to an infant: if the adult presents his/her genitals the infant will naturally grab. How this grabbing is to be distinguished by the infant from the thousands of other skin covered objects that he/she will grab through out his/her life remains a mystery to me.

Hypotheses:

Infants and children are not designed by evolution to avoid being involved in non-painful forms of sexual encounters which they are willing participants in. By "willing participant" all that is meant is not being forced to engage in the sexual act. The trauma that often follows sexual encounters with adults for children is caused by the reactions of the children's parents. There would be no trauma in the children if the parents and other role-models of said children saw sex with children as a routine part of growing up.

Experiments to Falsify:

(1): Take two appropriately large and randomized samples of infants and children. Have the control monitored by a brain imaging device while cleaned by their parents. Have the variable do the same only have researchers dressed in normal clothes do the cleaning as opposed to the parents. If there is a difference observed in the neurological behavior of these two groups which is larger than the difference between a group of children that are simply looking at their parents and looking at strangers, then there is likely a mechanism from birth which identifies sexual acts. All subjects must be sufficiently young so as to have no learned association with their genitals and sex.

(2): Find a closed population which has no concept of sex as a demonized act or of children as being too young to have sex with. Determine this by extensive interviews with the adult population designed to get them to be contradictory. After finding this population if it exists, show that the stability of those children which were involved in non-painful sexual acts with adults is lower than those children which were not involved. If this is accomplished it will suggest that the behaviors of parents of victims of molestation is not the source of the trauma caused in children after being molested.

Experiments to Verify:

(1): Setup the same control and variable as in (1) above. If we get the result that there is no significant difference between the neurological behavior of the control and the variable, then it becomes less likely that there is anything in children which allows them to tell the difference between non-painful acts of molestation, and cleaning of the genitals.

(2): Find a population as described in (2). Show that those individuals which engaged in sexual acts at a young age have no lower stability than those which did not. 

A Meme not a Gene:

If molestation is not inherently traumatic, why do we feel the need to protect our children from it? There are many possible reasons, but one of the most biological might be our jealousy. We are built to not let others have sex with loved ones, yes. But are we really biologically built to not let others have sex with our children? It'd be a strange adaptation to say the least. Why have children, and prevent them from reproducing? It might well be a side-effect of our evolved jealously. 

But more seems to be at play here then a confusion of jealousy. As my evidence for this I propose that you recall how salacious and downright offensive you found it when I mentioned that an infant would instinctively grab a genital if presented. It doesn't have to be your own infant in your mind to be repulsed by imagining the situation. It is a repulsive situation to imagine for almost anyone I have met that is not a pedophile, and even most pedophiles. If it is not our child we're imagining, just some random token child, and it is just some token child molester we are imagining, the image still repulses us greatly, which suggests that it does not come from biological design since our genetic fitness is not at all increased by worrying about the children of others.

We likely started demonizing pedophiles well after the development of language if the hypothesis stated above is correct. If trauma isn't caused in children from sexual acts with adults before learning about the taboo nature of sex, then it is likely the taboo nature of sex that causes such events to be traumatic. But sex is not taboo because of our genetic history, sex is taboo because of our memtic history.

Why the Meme is such a Success (Imagining Patient Zero):

Let's imagine a hypothetical culture which has demonized sex but doesn't really have an accepted attitude towards pedophiles. Suppose one parent catches another adult engaged in sexual behavior with his/her children. The parent, confused by and scared of sexual action, quickly pulls away the child while attacking the other adult and tells the child that he/she is not to do that anymore or go near that person. The child reacts negatively to this, now knowing that sex is demonic. We have all seen this sort of behavior before, if a child bumps his head and his/her parents say "Oh that's ok, come on, we gotta get going." in a lovely mommy voice the child is more likely to get up and keep on trucking. But if the parents react with "Oh God! Grab the ice pack, grab the ice pack!" yelling urgently, the child cries and may well act is if he/she is much more hurt than he/she really is.

When this hypothetical parent next sees his/her fellow parent friends he/she tells them of the event and how horrific it was for him/her, and how traumatic it was for his/her child. The other parents then warn their children of the strange man/woman that lured the first child and tell their own children never to go near that man/woman's house. The children of course need to find out why for themselves and go there anyway. Another child gets involved in acts of a sexual nature with the town pedophile. This catches the attention of a passerby, who by now knows of what goes on in that house, and how evil it is. This passerby alerts the others that it is happening again. At this point the town decides to do something about it. They lynch the pedophile. This becomes the talk of the town and of the local ruling government body.

Now all of the adults in the town know how to react to pedophilia: as if it would be a demonizing traumatic event for their children. Acting as such when one of their children is inevitably molested, causes that child to find it traumatic. News of the trauma it caused to the child spreads and the whole process is repeated, strengthening the believe that children become traumatized when molested. 

This thought experiment is likely not very much like what really happened to produce this meme in the first place. To actually understand how that happened we would have to trace the memetic evolution of our ancestors for much further than we have the ability to do now. But this hypothetical does at least give us a way of imagining how a belief like "Sexual acts with children and adults causes trauma in the children involved." might start off false and become truer as it becomes more widely accepted, and more widely accepted as it becomes truer. In the end holding that belief is going to cause more suffering in our children than if we didn't hold it provided the hypothesis above is correct. But we believe it anyway, and our moral judgements stray that way anyway, regardless of whether or not we have any benefit from the belief.

The true benefactor here is the meme itself. The meme of fearing and hating pedophiles need not be useful for us as a species, it needs only to be good at getting itself spread. Luckily for the meme, as it gets itself spread the belief associated with it becomes truer. This meme has a belief built in that is a self-fulfilling prophecy so that the more widespread the meme becomes the better its chances of replicating. It's a feedback loop, the meme predisposes us to act a certain way towards molested children, acting towards molested children this way makes them find the event traumatic, the observed trauma of the molested children enforces the meme.

Conclusion:

We can and do hold very basic moral attitudes as a culture which are completely unexamined. Even the most basic moral judgements that we make, like "pedophilia is wrong" are not on as firm of footing as we would like to believe them to be. But when we sharpen the issue and we are faced with the bluntness of the situation, things can become even more difficult. Our biases are very firmly rooted in us. Even I, who will tell you that I'm on the fence about the utility of demonizing pedophilia, am absolutely repulsed and ethically offended upon the thought of such an act. But I consider it important that we think sharply about the utility involved in such basic and unquestioned moral judgements and report our progress. If we find that those most basic moral judgements haven't been beneficial to us as a whole, we should start to wonder about whether or not ensuring utility really is the point of our moral system. Alternatively, our moral system might have little benefit to us and evolve only because it benefits the memes which it is. Our whole theory of ethics, might be the result of nothing more than the continued warfare of memes for our brains. Sometimes the memes convince us to adopt them by being beneficial, sometimes they just trick us into thinking they are right, and other times they make themselves true by the mere virtue of spreading themselves. This last class of memes we can call "self-proving memes" and it is this class of memes that the hypotheses above suggests the fearing and hating pedophiles meme belongs to. If that hypotheses is falsified by any of the suggested experiments or any other applicable experiment, we should still consider that the hypothesis has never even been suggested outside this text. Is this more likely because the hypotheses is so stupid, or because it is so rooted in us not to question such simple facts?

We may feel sympathy when we read about people killed for protesting in Syria, Bahrain, Libya, and other countries.  But tyranny isn't just something happening to unfortunate people somewhere else.  It's an existential risk to human civilization.

Civilization - even tribalism - relies on altruism.  Altruism is defined as cooperation that is not the happy convergence of interests of rational self-interested agents.  That happens too; but we don't call it altruism.  Altruism is, roughly, helping others without the expectation of reciprocation or cooperation.  And it happens because humans like helping other humans.

Altruism is probably mostly genetic.  It's an evolutionary adaptation that instills the desire to help others into a species.  Social pressure can install some amount of altruism; but it's my opinion that this would not work at all without a pre-existing genetic basis.  Many species exhibit altruism to a level at least as great as that in humans.  Some insects, which are incapable of feeling social pressure, are far more altruistic than humans.

Two theories for how this happens are kin selection and group selection.  Regardless of which of these you prefer, both of them have two important weaknesses:

• They are both very weak effects compared to selection for traits that benefit their organism directly.
• They require special social conditions, on society size (on the order of 10 members per society in the case of kin selection) and immigration/emigration rate (extremely low in both cases).

It's not known whether humans are still evolving, or have begun devolving due to lack of selective pressure.  But in the case of altruism, we can be sure:  Even if some selective pressure still exists, most humans today do not live under the necessary conditions for either kin selection or group selection.  Humans are living off their evolutionary capital of altruism.

Tyranny, whether it's that of Syria, Iran, North Korea, Nazi Germany, or the Soviet bloc under Stalin, aggressively selects against altruism.  The most-altruistic people were among the first executed in all those places.  They are the people being shot while protesting in Syria.  Social activism under such a government is rarely in your best self-interest.  Tyranny selects for self-interest; people who are willing to help the state oppress others are given opportunities for advancement.  And it removes altruistic genes quickly from the population, likely undoing hundreds of years of evolution every year.  Those genes will never be replaced.

I'm not too worried when this occurs over a few short months or years.  But when a people lives under these conditions for generations, you may end up with a large population deficient in altruistic genes.

There's no solution at that point short of gene therapy.  The population can stay in place, resulting in a society that is at best hopelessly mired in corruption and poverty, and at worst a danger to the rest of the world.  Or it can disperse, and dilute altruistic genes around the globe.

ADDED:  Knowing whether this is a real problem or not, would require learning something about how many genes are involved in altruism, and what their distribution in the population is.  A legitimate objection to what I wrote is that if genes for altruism are distributed so that killing less than 1% of the population would have a major impact on their abundance, then they probably weren't very important to begin with.  Although, sociopaths are only around 1% of the population, and they have a major impact on society.  I wonder how much work has been done in studying the maintenance of alleles for which you only need a few members of the population to have them?

http://eternal-earthbound-pets.com/Home_Page.html

Providing assurance that pets will be provided for in the event of Rapture.

Having thought it over, I'm OK with the ethics of this service.

Even though this was written by a current Less Wrong poster (hi, pdf23ds!), I don't think it has been posted here: Why and how to debate charitably (pg. 2, comments). (Edit: The original pdf23ds.net site has sadly been lost to entropy – Less Wrong poster MichaelBishop found a repost on commonsenseatheism.com. He also provides this summary version.)

I was linked to this article from a webcomic forum which had a low-key flamewar smouldering in the "Serious Business" section. (I will not link to it here; if you can tell from the description which forum it is, I would thank you not to link it either.) Three things struck me about it:

1. I have been operating under similar rules for years, with great success.
2. The participants in the flamewar on the forum where it was posted were not operating under these rules.
3. Less Wrong posters generally do operate under these rules, at least here.

The list of rules is on pg. 2 - a good example is the rule titled "You cannot read minds":

As soon as you find someone espousing seemingly contradictory positions, you should immediately suspect yourself of being mistaken as to their intent. Even if it seems obvious to you that the person has a certain intent in their message, if you want to engage them, you must respond being open to the possibility that where you see contradictions (or, for that matter, insults), none were intended. While you keep in mind what the person’s contradictory position seems to be, raise your standards some, and ask questions so that the person must state the position more explicitly—this way, you can make sure whether they actually hold it. If you still have problems, keep raising your standards, and asking more specific questions, until the person starts making sense to you.

If part of their position is unclear or ambiguous to you, say that explicitly. Being willing to show uncertainty is an excellent way to defuse the person’s, and your own, defensiveness. It also helps them to more easily understand which aspects of their position they are not making clear enough.

The less their position makes sense to you, the more you should rely on interrogative phrase and the less on declarative. Questions defuse defensiveness and are much more pointed and communicative than statements, because they force you to think more about the person’s arguments, and to really articulate what it about their position you most need clarification on. They help to keep the discussion moving, and help you to stop arguing past each other. Phrase the questions sincerely, and use as much of the person’s own reasoning (putting in the best light) as you can. This requires that you have a pretty good grasp on what the person is arguing—try to understand their position as well as you can. If it’s simply not coherent enough, the case may be hopeless.

In March, a user on Reddit emailed psychologist Philip Zimbardo (leader of the Stanford Prison Experiment) to arrange an "IAmA" interview. Zimbardo agreed to answer the top 5 questions from this thread. Yesterday his answers were posted here.

The chosen questions touched on research ethics, what he originally expected to learn from the experiment, the role of psychoactive drugs in society, reading recommendations and more.

After responding, Zimbardo posed a question of his own to Reddit:

I ask you: Is it good that the Milgram and Zimbardo studies were done, or wrong? Should they be allowed to be replicated with interesting variations (such as female guards and prisoners) if institutional guidelines are imposed and followed? Or is it better for society not to know about the nature of the "dark side" of human nature?

No, not psychoactive drugs: allergy drugs.

This is my attempt to come to grips with the idea of self-modification. I'm interested to know of any flaws folks might spot in this analogy or reasoning.

Gandhi wouldn't take a pill that would make him want to kill people. That is to say, a person whose conscious conclusions agree with their moral impulses wouldn't self-modify in such a way that they no longer care about morally significant things. But, what about morally insignificant things? Specifically, is willingness to self-modify about X a good guide to whether X is morally significant?

A person with untreated pollen allergies cares about pollen; they have to. In order to have a coherent thought without sneezing in the middle of it, they have to avoid inhaling pollen. They may even perceive pollen as a personal enemy, something that attacks them and makes them feel miserable. But they would gladly take a drug that makes them not care about pollen, by turning off or weakening their immune system's response to it. That's what allergy drugs are for.

But a sane person would not shut off their entire immune system, including responses to pathogens that are actually attacking their body. Even if giving themselves an immune deficiency would stop their allergies, a sane allergy sufferer wouldn't do it; they know that the immune system is there for a reason, to defend against actual attacks, even if their particular immune system is erroneously sensitive to pollen as well as to pathogens.

My job involves maintaining computer systems. Like other folks in this sort of job, my team use an automated monitoring system that will send us an alert (by pager or SMS), waking us up at night if necessary, if something goes wrong with the systems. We want to receive significant alerts, and not receive false positives. We regularly modify the monitoring system to prevent false positives, because we don't like being woken up at night for no good reason. But we wouldn't want to turn off the monitoring system entirely; we actually want to receive true alerts, and we will take action to refine our monitoring system to deliver more accurate, more timely true alerts — because we would like to improve our systems to make them fail less often. We want to win, and false positives or negatives detract from winning.

Similarly, there are times when we conclude that our moral impulses are incorrect: that they are firing off "bad! evil! sinful!" or "good! virtuous! beneficent!" alerts about things that are not actually bad or good; or that they are failing to fire for things which are. Performing the requisite Bayesian update is quite difficult: training yourself to feel that donating to an ineffective charity is not at all praiseworthy, or that it can be morally preferable to work for money and donate it, than to volunteer; altering the thoughts that come unbidden to mind when you think of eating meat, in accordance with a decision that vegetarianism is or is not morally preferable; and so on.

A sane allergy sufferer wants to update his or her immune system to make it stop having false positives, but doesn't want to turn it off entirely; and may want to upgrade its response sometimes, too. A sane system administrator wants to update his or her monitoring tools to make them stop having false positives, but doesn't want to turn it off entirely; and sometimes will program new alerts to avoid false negatives. There is a fact of the matter of whether a particular particle is innocuous pollen or a dangerous pathogen; there is a fact of the matter of whether a text message alert coincides with a down web server; and this fact of the matter explains exactly why we would or wouldn't want to alter our immune system or our servers' monitoring system.

The same may apply to our moral impulses: to decide that something is morally significant is, if we are consistent, equivalent to deciding that we would not self-modify to avoid noticing that significance; to decide that it is morally significant is equivalent to deciding that we would self-modify to notice it more reliably.

EDIT: Thanks for the responses. After mulling this over and consulting the Sequences, it seems that the kind of self-modification I'm talking about above is summed up by the training of System 1 by System 2 discussed waaaaay back here. Self-modification for FAI purposes is a level above this. I am only an egg.

http://www.rollingstone.com/politics/news/the-stoner-arms-dealers-20110316?print=true

The torture vs. dust specks quandary is a canonical one to LW.  Off the top of my head, I can't remember anyone suggesting the reversal, one where the arguments taken by the hypothetical are positive and not negative.  I'm curious about how it affects people's intuitions.  I call it - as the title indicates - "Sublimity vs. Youtube¹".

Suppose the impending existence of some person who is going to live to be fifty years old whatever you do².  She is liable to live a life that zeroes out on a utility scale: mediocre ups and less than shattering downs, overall an unremarkable span.  But if you choose "sublimity", she's instead going to live a life that is truly sublime.  She will have a warm and happy childhood enriched by loving relationships, full of learning and wonder and growth; she will mature into a merrily successful adult, pursuing meaningful projects and having varied, challenging fun.  (For the sake of argument, suppose that the ripple effects of her sublime life as it affects others still lead to the math tallying up as +(1 sublime life), instead of +(1 sublime life)+(various lovely consequences).)

Or you can choose "Youtube", and 3^^^3 people who weren't doing much with some one-second period of their lives instead get to spend that second watching a brief, grainy, yet droll recording of a cat jumping into a box, which they find mildly entertaining.

Sublimity or Youtube?

¹The choice in my variant scenario of "watching a Youtube video" rather than some small-but-romanticized pleasure ("having a butterfly land on your finger, then fly away", for instance) is deliberate.  Dust specks are really tiny, and there's not much automatic tendency to emotionally inflate them.  Hopefully Youtube videos are the reverse of that.

²I'm choosing to make it an alteration of a person who will exist either way to avoid questions about the utility of creating people, and for greater isomorphism with the "torture" option in the original.

(Apologies to RSS users: apparently there's no draft button, but only "publish" and "publish-and-go-back-to-the-edit-screen", misleadingly labeled.)

You have a button. If you press it, a happy, fulfilled person will be created in a sealed box, and then be painlessly garbage-collected fifteen minutes later. If asked, they would say that they're glad to have existed in spite of their mortality. Because they're sealed in a box, they will leave behind no bereaved friends or family. In short, this takes place in Magic Thought Experiment Land where externalities don't exist. Your choice is between creating a fifteen-minute-long happy life or not.

Do you push the button?

I suspect Eliezer would not, because it would increase the death-count of the universe by one. I would, because it would increase the life-count of the universe by fifteen minutes.

Actually, that's an oversimplification of my position. I actually believe that the important part of any algorithm is its output, additional copies matter not at all, the net utility of the existence of a group of entities-whose-existence-constitutes-utility is equal to the maximum of the individual utilities, and the (terminal) utility of the existence of a particular computation is bounded below at zero. I would submit a large number of copies of myself to slavery and/or torture to gain moderate benefits to my primary copy.

(What happens to the last copy of me, of course, does affect the question of "what computation occurs or not". I would subject N out of N+1 copies of myself to torture, but not N out of N. Also, I would hesitate to torture copies of other people, on the grounds that there's a conflict of interest and I can't trust myself to reason honestly. I might feel differently after I'd been using my own fork-slaves for a while.)

So the real value of pushing the button would be my warm fuzzies, which breaks the no-externalities assumption, so I'm indifferent.

But nevertheless, even knowing about the heat death of the universe, knowing that anyone born must inevitably die, I do not consider it immoral to create a person, even if we assume all else equal.

Just noticed that Less Wrong has apparently not yet linked to Bostrom & Yudkowsky's new paper for the forthcoming Cambridge Handbook of Artificial Intelligence, entitled "The Ethics of Artificial Intelligence." Enjoy.

Below is part 2 of the first draft of my book Ethics and Superintelligence. Your comments and constructive criticisms are much appreciated.

This is not a book for a mainstream audience. Its style is that of contemporary Anglophone philosophy. Compare to, for example, Chalmers' survey article on the singularity.

Bibliographic references and links to earlier parts are provided here.

Part 2 is below...

***

Late in the Industrial Revolution, Samuel Butler (1863) worried about what might happen when machines become more capable than the humans who designed them:

…we are ourselves creating our own successors; we are daily adding to the beauty and delicacy of their physical organisation; we are daily giving them greater power and supplying by all sorts of ingenious contrivances that self-regulating, self-acting power which will be to them what intellect has been to the human race. In the course of ages we shall find ourselves the inferior race.

…the time will come when the machines will hold the real supremacy over the world and its inhabitants…

By the time of the computer, Alan Turing (1950) realized that machines will one day be capable of genuine thought:

I believe that at the end of the century…  one will be able to speak of machines thinking without expecting to be contradicted.

Turing (1951/2004) concluded:

…it seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers... At some stage therefore we should have to expect the machines to take control…

All-powerful machines are a staple of science fiction, but one of the first serious arguments that such a scenario is likely came from the statistician I.J. Good (1965):

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion”, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

Vernor Vinge (1993) called this future event the “technological singularity.” Though there are several uses of the term “singularity” in futurist circles (Yudkowky 2007), I will always use the term to refer to Good’s predicted intelligence explosion.

David Chalmers (2010) introduced another terminological convention that I will borrow:

Let us say that AI is artificial intelligence of human level or greater (that is, at least as intelligent as an average human). Let us say that AI+ is artificial intelligence of greater than human level (that is, more intelligent than the most intelligent human). Let us say that AI++ (or superintelligence) is AI of far greater than human level (say, at least as far beyond the most intelligent human as the most intelligent human is beyond a mouse).

With this in place, Chalmers formalized Good’s argument like so:

1.     There will be AI (before long, absent defeaters).

2.     If there is AI, there will be AI+ (soon after, absent defeaters).

3.     If there is AI+, there will be AI++ (soon after, absent defeaters).

4.     Therefore, there will be AI++ (before too long, absent defeaters).

I will defend Chalmers’ argument in greater detail than he has, using “before long” to mean “within 150 years,” using “soon after” to mean “within two decades,” and using “before too long” to mean “within two centuries.” My definitions here are similar to Chalmers’ definitions, but more precise.

Following Chalmers, by “defeaters” I mean “anything that prevents intelligent systems (human or artificial) from manifesting their capacities to create intelligent systems.” Defeaters include “disasters, disinclination, and active prevention.”

Disasters include catastrophic events that would severely impede scientific progress, such as supervolcano eruption, asteroid impact, cosmic rays, climate change, pandemic, nuclear war, biological warfare, an explosion of nanotechnology, and so on. The risk of such disasters and others are assessed in Bostrom & Cirkovic (2008).

Disinclination refers to a lack of interest in developing AI of human-level general intelligence. Given the enormous curiosity of the human species, and the power that human-level AI could bring its creators, I think long-term disinclination is unlikely.

Active prevention of the development of human-level artificial intelligence has already been advocated by Thomas Metzinger (2004), though not because of the risk to humans. Rather, Metzinger is concerned about the risk to artificial agents. Early AIs will inevitably be poorly designed, which could lead to enormous subjective suffering for them that we cannot predict. One might imagine an infant from near Cherynobl whose parts are so malformed by exposure to nuclear radiation during development that its short existence is a living hell. In working toward human-level artificial intelligence, might we be developing millions of internally malformed beings that suffer horrible subjective experiences but are unable to tell us so?

It is difficult to predict the likelihood of the active prevention of AI development, but the failure of humanity to halt the development of ever more powerful nuclear weapons (Norris & Kristensen 2009) – even after tasting their destructive power – does not inspire optimism.

Later, we will return to consider these potential defeaters again. For now, let us consider the premises of Chalmers’ argument.

***

As previously announced, I plan to post the first draft of the book, Ethics and Superintelligence, in tiny parts, to the Less Wrong discussion area. Your comments and constructive criticisms are much appreciated.

This is not a book for a mainstream audience. Its style is that of contemporary Anglophone philosophy. Compare to, for example, Chalmers' survey article on the singularity.

Bibliographic references are provided here.

This "part 1" section is probably the only part of which I will post revision to Less Wrong. Revisions of further parts of the book will probably not appear publicly until the book is published.

Revised part 1 below....

1. The technological singularity is coming soon.

Every year, computers surpass human abilities in new ways. A program written in 1956 was able to prove mathematical theorems, and found a more elegant proof for one of them than Russell and Whitehead had given in Principia Mathematica (MacKenzie 1995). By the late 1990s, “expert systems” had surpassed human ability in a wide range of tasks.[i] In 1997, IBM’s Deep Blue defeated the reigning World Chess Champion Garry Kasparov (Campbell et al. 2002). In 2011, IBM’s Watson beat the best human players at a much more complicated game: Jeopardy! (Someone, 2011). Recently, a robot scientist was programmed with our scientific knowledge about yeast, then posed its own hypotheses, tested them, and assessed the results. It answered a question about yeast that had baffled human scientists for 150 years (King 2011).

Many experts think that human-level general intelligence may be created within this century.[ii] This raises an important question. What will happen when an artificial intelligence (AI) surpasses human ability at designing artificial intelligences?

I.J. Good (1965) speculated that such an AI would be able to improve its own intelligence, leading to a positive feedback loop of improving intelligence – an “intelligence explosion.” Such a machine would rapidly become intelligent enough to take control of the internet, use robots to build itself new hardware, do science on a massive scale, invent new computing technology and energy sources, or achieve similar dominating goals. As such, it could be humanity’s last invention (Bostrom 2003).

Humans would be powerless to stop such a “superintelligence” (Bostrom 1998) from accomplishing its goals. Thus, if such a scenario is at all plausible, then it is critically important to program the goal system of this superintelligence such that it does not cause human extinction when it comes to power.

Success in that project could mean the difference between a utopian solar system of unprecedented harmony and happiness, and a solar system in which all available matter (including human flesh) has been converted into parts for a planet-sized computer built to solve difficult mathematical problems.[iii]

The technical challenges of designing the goal system of such a superintelligence are daunting.[iv] But even if we can solve those problems, the question of which goal system to give the superintelligence remains. It is at least partly a question of philosophy – a question of ethics.

***

In this chapter I argue that a single, powerful superintelligence - one variety of what Bostrom (2006) calls a “singleton" - is likely to arrive within the next 200 years unless a worldwide catastrophe drastically impedes scientific progress.

The singleton will produce very different future worlds depending on which normative theory is used to design its goal system. In chapter two, I survey many popular normative theories, and conclude that none of them offer an attractive basis for designing the motivational system of a machine superintelligence.

Chapter three reformulates and strengthens what is perhaps the most developed plan for the design of the singleton’s goal system ­– Eliezer Yudkowsky’s (2004) “Coherent Extrapolated Volition.” Chapter four considers some outstanding worries about this plan.

In chapter five I argue that we cannot decide how to design the singleton’s goal system without considering meta-ethics, because normative theory depends on meta-ethics. The next chapter argues that we should invest little effort in meta-ethical theories that do not fit well with our emerging reductionist picture of the world, just as we quickly abandon scientific theories that don’t fit the available scientific data. I also identify several meta-ethical positions that I think are good candidates for abandonment.

But the looming problem of the technological singularity requires us to have a positive theory, too. Chapter seven proposes some meta-ethical claims about which I think naturalists should come to agree. In the final chapter, I consider the implications of these meta-ethical claims for the design of the singleton’s motivational system.

***

From Mike Darwin's new blog:

DCD has lead to a fracture within the medical community [7,8] wherein some centers, such as the University of Pittsburgh Medical Center, have taken patients who want ventilator support withdrawn, placed femoral cannulae under local (spinal) anesthesia, turned off the ventilator after effectively anesthetizing the patient, waited until the patient’s heart stops, and then restarted circulation with CPB. They also, of course, give paralytic neuromuscular blocking drugs (as is routine in all visceral organ retrieval) to prevent the thoracoabdominal incision, and the terminal drop in blood pressure (when the organs are removed), from causing muscle vesiculations (twitching) or actual limb movement as a result of stimulation of the nocioceptive pathways in the spinal cord (pain is a local phenomenon first and a central nervous system one secondly with the process proceeding up the spinal cord to the brain). [9,10]

To be blunt, this procedure resulted in all hell breaking out. [11,12,13] Bioethicists, such James Bernat and Leslie Whetstine, accused the surgeons and neurologists involved in this undertaking of every ethical evil, including homicide.[14,15] A compromise position is to restore circulation in the body using a special balloon-tipped aortic catheter that prevents ‘all ‘ flow to the brain. This results in a ‘resolution’ to the ‘paradox’ of removing organs from a patient with a ‘viable, or potentially viable brain.’ Of course, from our perspective as cryonicists, this whole exercise is nothing more or less than a procedural contortion designed to avoid confronting the reality that death is not a binary condition, and that if you are going to allow people to withdraw from medical care they no longer want, and that they (rightfully) consider an assault, then the corollary to that is that they also get to decide when they are dead. [16] That means that they have the perfect right to ask for, and receive a treatment (i.e., in the presence of informed consent) whereby they are anesthetized, cooled, subjected to blood washout, and their organs removed – at which point they are indeed DEAD, in the sense that their non-functional condition is now irreversible, or not going to be reversed, because they do not want it to be. When, exactly, they become irrecoverable from an information-theoretic standpoint is irrelevant, because they don’t want to be recovered, and no technology currently exists that will allow them to be recovered.

We, as cryonicists, could argue that if such patients were cryopreserved, they might possibly be recovered in the future. But if they do not want cryopreservation, then they are dead when they say they are dead, and when they meet the current medico-legal definition of cardiorespiratory death (i.e., no heartbeat or breathing and no prospect of their resuming). The medical response to this fairly straightforward situation has been, as expected, convoluted and irrational, and profoundly dangerous to cryonics. The recent paper “Clarifying the paradigm for the ethics of donation and transplantation: Was ‘dead’ really so clear before organ donation?” [17] is an excellent window into current medical policy, not just on the issue of DCD, but on the application of any kind of circulatory support to patients who have been pronounced dead on the basis of clinical (cardiac) criteria.  This article is one of the most cited in current DCD debates, and the closing sentence in its abstract says it all (emphasis mine):

“Criticism of controlled DCD on the basis of violating the dead donor rule, where autoresuscitation has not been described beyond 2 minutes, in which life support is withdrawn and CPR is not provided, is not valid. However, any post mortem intervention that reestablishes brain blood flow should be prohibited. “In comparison to traditional practice, organ donation has forced the clarification of the diagnostic criteria for death and improved the rigour of the determinations.”[17]

...

The UK has already adopted standards for determining and pronouncing death that expressly prohibit the application of CPR, or any modalities that restore flow to the brain or conserve brain viability. I have made inquiries, and been informed that failure to follow these Guidelines would be a serious breach of professional conduct, resulting in any licensed person being struck off; and that such action would very likely constitute a criminal act in the UK, as well (prosecution to be at the discretion of law enforcement and the prosecutor). [21]

The whole point of cryonics -- not to put too fine a point on it -- is to conserve brain viability, in the sense of keeping as much of the brain in as close to a viable state as possible.

ETA: Mike has confirmed that the UK law applies to non organ donors. He also has stated that new changes have been made to the Uniform Anatomical Gift Act (a sort of template by which state laws are drafted) which are likely to be similar in nature, in the US.

Earlier, I provided an overview of formal epistemology, a field of philosophy highly relevant to the discussions on Less Wrong. Today I do the same for another branch of philosophy: the philosophy of artificial intelligence (here's another overview).

Some debate whether machines can have minds at all. The most famous argument against machines achieving general intelligence comes from Hubert Dreyfus. The most famous argument against the claim that an AI can have mental states is John Searle's Chinese Room argument, to which there are many replies. The argument comes in several variations. Most Less Wrongers have already concluded that yes, machines can have minds. Others debate whether machines can be conscious. 

There is much debate on the significance of variations on the Turing Test. There is also lots of interplay between artificial intelligence work and philosophical logic. There is some debate over whether minds are multiply realizable, though most accept that they are. There is some literature on the problem of embodied cognition - human minds can only do certain things because of their long development; can these achievements be replicated in a machine written "from scratch"?

Of greater interest to me and perhaps most Less Wrongers is the ethics of artificial intelligence. Most of the work here so far is on the rights of robots. For Less Wrongers, the more pressing concern is that of creating AIs that behave ethically. (In 2009, robots programmed to cooperate evolved to lie to each other.) Perhaps the most pressing is the need to develop Friendly AI, but as far as I can find, no work on Good's intelligence explosion singularity idea has been published in a major peer-reviewed journal except for David Chalmers' "The Singularity: A Philosophical Analysis" (Journal of Consciousness Studies 17: 7-65). The next closest thing may be something like "On the Morality of Artificial Agents" by Floridi & Sanders.

Perhaps the best overview of the philosophy of artificial intelligence is chapter 26 of Russell & Norvig's Artificial Intelligence: A Modern Approach.

Related to Your Rationality is My Business

Among religious believers in the developed world, there is something of a hierarchy in terms of social tolerability. Near the top are the liberal, nonjudgmental, frequently nondenominational believers, of whom it is highly unpopular to express disapproval. At the bottom you find people who picket funerals or bomb abortion clinics, the sort with whom even most vocally devout individuals are quick to deny association.

Slightly above these, but still very close to the bottom of the heap, are proselytizers and door to door evangelists. They may not be hateful about their beliefs, indeed many find that their local Jehovah’s Witnesses are exceptionally nice people, but they’re simply so annoying. How can they go around pressing their beliefs on others and judging people that way?

I have never known another person to criticize evangelists for not trying hard enough to change others’ beliefs.

In the novel Life Artificial I use the following assumptions regarding the creation and employment of AI personalities.

1. AI is too complex to be designed; instances are evolved in batches, with successful ones reproduced
2. After an initial training period, the AI must earn its keep by paying for Time (a unit of computational use)

(Note: PDA := AI, Sticky := human)

The second fitness gradient is based on economics and social considerations: can an AI actually earn a living? Otherwise it gets turned off.

As a result of following this line of thinking, it seems obvious that after the initial novelty wears off, AIs will be terribly mistreated (anthropomorphizing, yeah).

It would be very forward-thinking to begin to engineer barriers to such mistreatment, like a PETA for AIs. It is interesting that such an organization already exists, at least on the Internet: ASPCR

Premise:  There exists a community whose top-most goal is to maximally and fairly fulfill the goals of all of its members.  They are approximately as rational as the 50th percentile of this community.  They politely invite you to join.  You are in no imminent danger.

Do you:

• Join the community with the intent to wholeheartedly serve their goals
• Join the community with the intent to be a net positive while serving your goals
• Politely decline with the intent to trade with the community whenever beneficial
• Politely decline with the intent to avoid the community
• Join the community with the intent to only do what is in your best interest
• Politely decline with the intent to ignore the community
• Join the community with the intent to subvert it to your own interest
• Enslave the community
• Destroy the community
• Ask for more information, please

Premise:  The only rational answer given the current information is the last one.

What I’m attempting to eventually prove The hypothesis that I'm investigating is whether "Option 2 is the only long-term rational answer". (Yes, this directly challenges several major current premises so my arguments are going to have to be totally clear.  I am fully aware of the rather extensive Metaethics sequence and the vast majority of what it links to and will not intentionally assume any contradictory premises without clear statement and argument.)

It might be an interesting and useful exercise for the reader to stop and specify what information they would be looking next for before continuing.  It would be nice if an ordered list could be developed in the comments.

Obvious Questions:

<Spoiler Alert>

1. What happens if I don’t join?
2. What do you believe that I would find most problematic about joining?
3. Can I leave the community and, if so, how and what happens then?
4. What are the definitions of maximal and fairly?
5. What are the most prominent subgoals?/What are the rules?

It has been very enjoyable to post on LW [1, 2] and I have learned a lot from the discussions with other members, for which I am very thankful. But unfortunately, judging by my karma score which is on the same level of Kiwiallegiance and the Jewelry spammer, my opinions are not appreciated and I frequently receive the following message when posting a new comment:

You are trying to submit too fast. try again in x_n minutes.

When I press the submit button after x₁ + 1 minutes, I'm told to wait another x₂ minutes. So commenting has become more and more frustating, and I don't want to continue burden the LW members with the heavy tast of down-voting me. But on the other hand, I still can't find any flaw in my argumentation despite many rebuttals. Maybe I am too ignorant, maybe I am on something. So I'll give myself a last try.

You might explain my desire for pancakes as a neuronal process, as a mental function biologically evolved, but it does not capture the meaning of "desire". The concept of meaning itself has no meaning in the natural world, but it has a meaning to us, to the rational mind.

As much as we cannot explain what the natural world "really" is, since we cannot see what is behind the physical reality (unless you are an idealistic Platonist), we cannot explain what morality and values "really" are. We can only describe them using scientific theories or normative theories, respectively.

The following is destilled from the comment section of an earlier post.

Definitions

absolute and universal: Something that applies to everything and every mind.

morality (moral world): A logically consistent system of normative theories.

reality (natural world): A logically consistent system of scientific (natural) theories.

normative theory: (Almost) any English sentence in imperative or including the word "should", "must", "to be allowed to" as the verb or equivalent construction, in contrast to descriptive theories.

mind: A mind is an intelligence that has values, desires and dislikes.

moral perception: Analogous to the sensory perceptions, a moral perception is the feeling of right and wrong.

Assumptions

A normative sentence arises as a result of the mind processing its values, desires and dislikes.

Ideas exist independently from the mind. Numbers don't stop to exist just because HAL dies.

Statement

In our everyday life, we don't question the reality, due to our sensory perception. We have moral perception as much as we have a sensory perception, therefore why should we question morality?

If you believe that the natural world is absolute and universal, then there is -- I currently think -- no good reason to doubt the existence of an absolute and universal moral world.

A text diagram for illustration

Absolute moralily

The absolute moral world, I am talking about, does encompass everything, including AI and alien intelligence. It does not mean that alien intelligence will behave similarly to us. Different moral problems require different solutions, as much as different objects behave differently according to the same physical theories. Objects in vacuum behave differently than in the atmosphere. Water behaves differently than ice, but they are all governed by the same physics, so I assume.

An Edo-ero samurai and a Wall Street banker may behave perfectly moral even if they act differently to the same problem due to the social environment. Maybe it is perfectly moral for AIs to kill and annihilate all humans, as much as it is perfectly possible that 218 of Russell's teapots are revolving around Gliese 581 g.

The intersubjective consensus

There are different sets of theories regarding the natural world: the biblical view, the theories underlying TCM, the theories underlying homeopathy, the theories underlying chiropractise and the scientific view. Many of them contradict each other. The scientific view is well-established because there is an intersubjective consensus on the usefulness of the methodology.

The methods used in moral discussions are by far not so rigidly defined as in science; it's called civil discourse. The arguments must be logical consistent and the outcomes and conclusions of the normative theory must face the empirical challenge, i.e. if you can derive from your normative theories that it is permissible to kill innocent children without any benefits, then there is probably something wrong.

Using this method, we have done quite a lot so far. We have established the UN Human Rights Charta, we have an elaborated system of international law, law itself being a manifestation of morality (denying the fact, that law is based on morality is like saying that technology isn't based on science).

Not everyone might agree and some say, "I think that chattel slavery is perfectly moral." And there are people who think that praying to an almighty pasta monster and dressing up as pirates will cure all the ills of the world. Does that mean that there is no absolute reality? Maybe.

Conclusion

As long as we have values, desires, dislikes and make judgements (which all of us do and which maybe is a defining characteristic of the human being beyond the biological basics), if we want to put these values into a logical consistent system, and if we believe that other minds with moral perception exist, then we have an absolute moral world.

So if we stop having any desires and stop making any judgements, that is if we lack any moral perception, then we may still believe in morality, as much as an agnostic won't deny the existence of God, but it would be totally irrelevant to us.

To the same degree, if someone lacks all the sensory perception, then the natural world becomes totally irrelevant to him or her.

Level 1: Algorithm-based Intelligence

An intelligence of level 1 acts on innate algorithms, like a bacterium that survives using inherited mechanisms. 

Level 2: Goal-oriented Intelligence

An intelligence of level 2 has an innate goal. It develops and finds new algorithms to solve a problem. For example, the paperclip maximizer is a level-2 intelligence.

Level 3: Philosophical Intelligence

An intelligence of level 3 has neither any preset algorithms nor goals. It looks for goals and algorithms to achieve the goal. Ethical questions are only applicable to intelligence of level 3.