You know, the funny thing is, there are transhumanist themes in the drafts and deleted materials for Evangelion. For example, parsing the SEELE discussions in the EoE draft and comments by Gainaxers, one has the impression that originally the plan of Gendo and Yui was to upload humans into immortal Evas so they could colonize other worlds, the human body being too frail and short-lived for space travel! (Like many of the explanations and details, I think they were cut by Anno to focus on the psychology themes that interested him more.)

I'm curious. Have you ever lost a loved one due to someone else's actions? The closest experience I have to this is a cousin who was killed about a year ago by a speeding driver. My cousin Brandon wasn't that old. He hadn't been a great student in highschool but had really shaped up and become a lot more responsible in college. Brandon was working to become a chef, something he was clearly good at and clearly enjoyed. My cousin was on his bike and never even saw the car. He had on a helmet. It saved his life, for a few days. His grandmother, my aunt, was on an airplane flight when the accident happened. She was on her way to the funeral of another relative who had killed himself. She found out about the accident as her plane taxied to the gate.

At first, after a few days in the hospital it seemed that Brandon was going to make it. Then he took a sudden turn for the worst and his organs started to fail. The end was so sudden that some of my relatives saw in their inboxes the email update saying that Brandon wasn't like to make it right under the email saying he had died.

Then, it turned out that the driver of the car had a history of speeding problems. He received in a year in jail for vehicular homicide. A small compensation for the entire life Brandon had in front of him.

If someone came up to me, and gave me the choice of making that driver die a slow painful, agonizing death I'd probably say yes. It would be wrong. Deeply wrong. But the emotion is that strong; I don't know if I could override it.

But I can still understand that that's wrong. The driver was an aging Vietnam vet with a history of medical problems. He had little family. He was so distraught over what happened that when initially put in jail before the trial, there was worry that he might kill himself. He seems to be an old, lonely, broken man. Harming him accomplishes little. And yet, despite all that, the desire to see him suffer still burns deeply within me.

How much more would I feel if I thought that someone had killed a relative, or even my own child? And if the court had repeatedly agreed and told me that that was the guilty person. How could I ever emotionally acknowledge that I had been after the wrong person, that not only had I persecuted the wrong person, but the person who had done this terrible deed was still out there, and free? I'd like to believe that I'm a rational person so that I could make that acknowledgment. But the fact that even when it is just a cousin I still deeply desire someone to suffer in ways that help no one at all... I doubt I could do it.

To call the Kerchers evil or their desires evil is a deep failure of empathy.

We're not in disagreement about that. But your assumption that emotions are necessary for goals to be formed is still an untested one.

There's a relevant factoid that's come up here on LW a few times before: Apparently, people with significant brain damage to their emotional centers are unable to make choices between functionally near-identical things, such as different kinds of breakfast cereal. But, interestingly, they get stuck when trying to make those choices - implying that they do attempt to e.g. acquire cereal in the first place; they're not just lying in a bed somewhere staring at the ceiling, and they don't immediately give up the quest to acquire food as unimportant when they encounter a problem.

It would be interesting to know the events that lead up to the presented situation; it would be interesting to know whether people with that kind of brain damage initiate grocery-shopping trips, for example. But even if they don't - even if the grocery trip is the result of being presented with a fairly specific list, and they do otherwise basically sit around - it seems to at least partially disprove your 'standby mode' theory, which would seem to predict that they'd just sit around even when presented with a grocery list and a request to get some shopping done.

Clippy is usually brought up as a most dangerous AI that we should avoid creating at all costs, yet what's the point of creating copies of us and tile the universe with them? how is that different than what clippy does?

That's an easy one. I value humans, I don't value paperclips.

Shouldn't we focus on engineered/controlled value drift rather than preventing it entirely?

According to EY's CEV document, CEV does this. It lets/makes our values drift in the way we would want them to drift.

without a body the brain won't 'work', the brain is very much linked to the rest of the body, the fiction that we only need the head to 'reanimate' a person back to normal is just that, fiction.

wei Dai:"rebuilding/simulating the body to the level of detail needed to support cognition" yes,but how complex is the nervous system? which wire connects to which, or is that not important? seems to me that you're oversimplifying..

One marker to watch out for is a kind of selection effect.

In some fields, only 'true believers' have any motivation to spend their entire careers studying the subject in the first place, and so the 'mainstream' in that field is absolutely nutty.

Case examples include philosophy of religion, New Testament studies, Historical Jesus studies, and Quranic studies. These fields differ from, say, cryptozoology in that the biggest names in the field, and the biggest papers, are published by very smart people in leading journals and look all very normal and impressive but those entire fields are so incredibly screwed by the selection effect that it's only "radicals" who say things like, "Um, you realize that the 'gospel of Mark' is written in the genre of fiction, right?"

More specifically it is completely rational to use that argument against theists, because one of their arguments for god is that the world is too complex not to have been designed; so in that circumstance you are just pointing out that their claim is just pushing the complexity back one step. If the world is so complex that it needs a designer, then so is god.

If they can't stop students from using Wikipedia, pretty soon schools will be reduced from teaching how to gather facts, to teaching how to think!

Well, I'm not sure how far that advances things, but a possible failure mode -- or is it? -- of a Friendly AI occurs to me. In fact, I foresee opinions being divided about whether this would be a failure or a success.

Someone makes an AI, and intends it to be Friendly, but the following happens when it takes off.

It decides to create as many humans as it can, all living excellent lives, far better than what even the most fortunate existing human has. And these will be real lives, no tricks with simulations, no mere tickling of pleasure centres out of a mistaken idea of real utility. It's the paradise we wanted. The only catch is, we won't be in it. None of these people will be descendants or copies of us. We, it decides, just aren't good enough at being the humans we want to be. It's going to build a new race from scratch. We can hang around if we like, it's not going to disassemble us for raw material, but we won't be able to participate in the paradise it will build. We're just not up to it, any more than a chimp can be a human.

It could transform us little by little into fully functional members of the new civilisation, maintaining continuity of identity. However, it assures us, and our proof of Friendliness assures us that we can believe it, the people that we would then be would not credit our present selves as having made any significant contribution to their identity.

Is this a good outcome, or a failure?