I recommend reading this sequence.

Thanks for recommending.

Suffice it to say that you are wrong, and power does not bring with it morality.

I have never assumed that "power brings with it morality" if we with power mean limited power. Some superhuman AI might very well be more immoral than humans are. I think unlimited power would bring with it morality. If you have access to every single particle in the universe and can put it wherever you want, and thus create whatever is theoretically possible for an almighty being to create, you will know how to fill all of spacetime with the largest possible amount of happiness. And you will do that, since you will be intelligent enough to understand that that's what gives you the most happiness. (And, needless to say, you will also find a way to be the one to experience all that happiness.) Given hedonistic utilitarianism, this is the best thing that could happen, no matter who got the unlimited power and what was initially the moral standards of that person. If you don't think hedonistic utilitarianism (or hedonism) is moral, it's understandable that you think a world filled with the maximum amount of happiness might not be a moral outcome, especially if achieving that goal took killing lots of people against their will, for example. But that alone doesn't prove I'm wrong. Much of what humans think to be very wrong is not in all circumstances wrong. To prove me wrong, you have to either prove hedonism and hedonistic utilitarianism wrong first, or prove that a being with unlimited power wouldn't understand that it would be best for him to fill the universe with as much happiness as possible and experience all that happiness.

a happy person doesn't hate.

What is your support for this claim?

Observation.

Now this is the $64 google-illion question!

I don't agree that the null hypothesis: take the ring and do nothing with it is evil. My definition of evil is coercion leading to loss of resources up to and including loss of one's self. Thus absolute evil is loss of one's self across humanity which includes as one use case humanity's extinction (but is not limited to humanity's extinction obviously because being converted into zimboes isn't technically extinction..)

Nobody can argue that the likes of Gaddafi exist in the human population: those who are interested in being the total boss of others (even thought they add no value to the lives of others) to the extent that they are willing to kill to maintain their boss position.

I would define these people as evil or with evil intent. I would thus state that I would like under no circumstances somebody like this to grab the ring of power and thus I would be compelled to grab it myself.

The conundrum is that I fit the definition of evil myself. Though I don't seek power to coerce as an end in itself I would like the power to defend myself against involuntary coercion.

So I see a Gaddafi equivalent go to grab the ring and I beat him to it.

What do I do next?

Well I can't honestly say that I have the right to kill the millions of Gaddafi equivalent but I think that on average they add a net negative to the utility of humanity.

I'm left, however, with the nagging suspicion that under certain circumstances, Gaddafi type figures might be beneficial to humanity as a whole. Consider: crowdsourcing the majority of political decisions would probably satisfy the average utility function of humanity. It's fair but not to everybody. We have almost such a system today (even though it's been usurped by corporations). But in times of crisis such as during war, it's more efficient to have rapid decisions made by a small group of "experts" combined with those who need to make ruthless decisions so we can't kill the Gaddafis.

What is therefore optimal in my opinion? I reckon I'd take all the Gaddafis off planet and put them in simulations to be recalled only at times of need and leave sanitized nice people zimbo copies of them. Then I would destroy the ring of power and return to my previous life before I was tempted to torture those who have done me harm in the past.

Xannon decides how much Zaire gets. Zaire decides how much Yancy gets. Yancy decides how much Xannon gets.

If any is left over they go through the process again for the remainder ad infinitum until an approximation of all of the pie has been eaten.

I agree that there is physical continuity from moment to moment in typical human existence, and that there is similar continuity with a slow transition to a nonhuman form. I agree that there is no such continuity with an instantaneous copy-and-destroy operation.

I understand that you consider that difference uniquely important, such that I continue living in the first case, and I don't continue living in the second case.

I infer that you believe in some uniquely important attribute to my self that is preserved by the first process, and not preserved by the second process.

I agree that if a person is being offered a choice, it is important for that person to understand the choice. I'm perfectly content to describe the choice as between the death of one body and the creation of another, on the one hand, and the continued survival of a single body, on the other. I'm perfectly content not to describe the latter process as the continuation of an existing life.

I endorse individuals getting to make informed choices about their continued life, and their continued existence as people, and the parameters of that existence. I endorse respecting both their stated wishes, and (insofar as possible) their volition, and I acknowledge that these can conflict given imperfect information about the world.

Do you have a problem with involuntary forced destructive scanning in order to upload individuals into some other substrate (or even a copied clone)?

Yes. As I say, I endorse respecting individuals' stated wishes, and I endorse them getting to make informed choices about their continued existence and the parameters of that existence; involuntary destructive scanning interferes with those things. (So does denying people access to destructive scanning.)

Do you consider a person who has lost much of their memory to be the same person?

It depends on what 'much of' means. If my body continues to live, but my memories and patterns of interaction cease to exist, I have ceased to exist and I've left a living body behind. Partial destruction of those memories and patterns is trickier, though; at some point I cease to exist, but it's hard to say where that point is.

What if such a person (who has lost much of their memory) then has a backed up copy of their memories from six months ago imprinted over top?

I am content to say I'm the same person now that I was six months ago, so if I am replaced by a backed-up copy of myself from six months ago, I'm content to say that the same person continues to exist (though I have lost potentially valuable experience). That said, I don't think there's any real fact of the matter here; it's not wrong to say that I'm a different person than I was six months ago and that replacing me with my six-month-old memories involves destroying a person.

What if it's someone else's memories: did they just die?

If I am replaced by a different person's memories and patterns of interaction, I cease to exist.

Scan a person destructively (with their permission). Keep their scan in storage on some static substrate. Then grow a perfectly identical clone of them (using "identical" to mean functionally indentical because we can't get exactly identical as discussed before). Copy the contents of the mindstates into that clone. How many deaths have taken place here?

Several trillion: each cell in my current body died. I continue to exist. If my clone ever existed, then it has ceased to exist.

Incidentally, I think you're being a lot more adversarial here than this discussion actually calls for.

There's a lot of decent SF on this theme. If you haven't read John Varley's Eight Worlds stuff, I recommend it; he has a lot of fun with this. His short stories are better than his novels, IMHO, but harder to find. "Steel Beach" isn't a bad place to start.

I can't personally say that I'm willing to come out and say definitively that the X is a red herring though it sounds like you are willing to do this.

I did not say the X is a red herring. If you believe I did, I recommend re-reading my comment.

The X is far from being a red herring; rather, the X is precisely what I was trying to elicit details about for a while. (As I said above, I no longer believe I can do so through further discussion.)

But I did say that identity of quantum states is a red herring.

As I said before, I conclude this from the fact that you believe you are the same person you were last year, even though your quantum states aren't identical. If you believe that X can remain unchanged while Y changes, then you don't believe that X depends on Y; if you believe that identity can remain unchanged while quantum states change, then you don't believe that identity depends on quantum states.

To put this another way: if changes in my quantum states are equivalent to my death, then I die constantly and am constantly replaced by new people who aren't me. This has happened many times in the course of writing this comment. If this is already happening anyway, I don't see any particular reason to avoid having the new person appear instantaneously in my mom's house, rather than having it appear in an airplane seat an incremental distance closer to my mom's house.

Other stuff:

• Yes, I would say that if the daughter cell is identical to the parent cell, then it doesn't matter that the parent cell died at the instant of budding.

• I would also say that it doesn't matter that the vast majority of the cells comprising me twenty years ago are dead, even though the cells currently comprising me aren't identical to the cells that comprised me then.

• I agree with you that if a person is perfectly duplicated and the original killed, then the original has been killed. (I would also say that the person was killed, which I think you would agree with. I would also say that the person survived, which I think you would not agree with.)

• I agree that volition is important for its own sake, but I don't understand what volition has to do with what we've thus far been discussing. If forcing the original to bud kills the original, then it does so whether the original wants to die or not. If it doesn't kill the original, then it doesn't, whether the original wants to die or not. It might be valuable to respect people's volition, but if so, it's for some reason independent of their survival. (For example, if they want to die, then respecting their volition is opposed to their survival.)

• A question for you: if someone wants to stop existing, and they destructively scan themselves, am I violating their wishes if I construct a perfect duplicate from the scan? I assume your answer is "no," since the duplicate isn't them; they stopped existing just as they desired.

It's as if people compartmentalize them and think about only one or the other at a time.

Or just disagree with a specific transhumanist moral (or interpretation thereof). If you are growing "too powerful too quickly" the right thing for an FAI (or, for that matter, anyone else) to do is to stop you by any means necessary. A recursively self improving PhilGoetz with that sort of power and growth rate will be an unfriendly singularity. Cease your expansion or we will kill you before it is too late.

Phil: an AI who is seeking resources to further it's own goals at the expense of everyone else is by definition an unfriendly AI.

The question is whether the PhilGoetz utility function, or the average human utility function, are better. Assume both are implemented in AIs of equal power. What makes the average human utility function "friendlier"? It would have you outlaw homosexuality and sex before marriage, remove all environmental protection laws, make child abuse and wife abuse legal, take away legal rights from women, give wedgies to smart people, etc.

Now consider this: I'd prefer the average of all human utility function over my maximized utility function even if it means I have less utility.

I don't think you understand utility functions.

If it's not fear what is your objection to having your heart broken?

The same objection I have to someone cutting off my little toe. It is painful and means that I'll forever be missing a part of myself. Not a big deal - just a minor to moderate negative outcome.

And how can you possibly take upon yourself the right to decide for everybody else?

You are responding to a straw man again - and I am rather surprised that you have been rewarded for doing so since it is rather insulting to attribute silly beliefs to people without cause. This is a complete reversal of what Wedrifid_2010 said. He vehemetly rejected thepokeduck's proposal that everyone should have their heart broken - because he found the idea of someone deciding that everyone else should have their heart broken abhorrent and presumptive.

Then, in the very comment you replied to, Wedrifid_2010 said:

Sure, if they are into that sort of thing I don't particularly care.

That is explicitly declaring no inclination toward controlling other people's self-heart-breaking impulses.