Folk at the Singularity Institute and the Future of Humanity Institute agree that it would probably (but unstably in the face of further analysis) be better to have brain emulations before de novo AI from an existential risk perspective (a WBE-based singleton seems more likely to go right than an AI design optimized for ease of development rather than safety). I actually recently gave a talk at FHI about the use of WBE to manage collection action problems such as Robin Hanson's "Burning the Cosmic Commons" and pressures to cut corners on safety of AI development, which I'll be putting online soon. One of the projects being funded by the SIAI Challenge Grant ending tonight is an analysis of the relationship between AI and WBE for existential risks.
However, the conclusion that accelerating WBE (presumably via scanning or neuroscience, not speeding up Moore's Law type trends in hardware) is the best marginal project for existential risk reduction is much less clear. Here are just a few of the relevant issues:
1) Are there investments best made far in advance with WBE or AI? It might be that the theory to build safe AIs cannot be rushed as much as institutions to manage WBEs,...
It would have been good to check this suggested post topic on an Open Thread, first - in fact I should get around to editing the FAQ to suggest this for first posts.
Perhaps the AI rules wisely and well, and can give us anything we want, "save relevance".
In addition to the retreads that others have pointed out on the upload safety issue, this is a retread of the Fun Theory Sequence:
http://lesswrong.com/lw/xy/the_fun_theory_sequence/
Also the way you phrased the above suggests that we build some kind of AI and then discover what we've built. The space of mind designs is very large. If we know what we're doing, we reach in and get whatever we specify, including an AI that need not steal our relevance (see Fun Theory above). If whoever first reaches in and pulls out a self-improving AI doesn't know what they're doing, we all die. That is why SIAI and FHI agree on at least wistfully wishing that uploads would come first, not the relevance thing. This part hasn't really been organized into proper sequences on Less Wrong, but see Fake Fake Utility Functions, the Metaethics sequence, and the ai / fai tags.
Yes, it's not 100% certain that a self-enhancing AI will overwhelm human resistance. One can construct scenarios where the process is halted in time. For example, the AI's cognition can be monitored, and everything halted if a goal like "prevent the stop button from being pressed" ever crosses its mind. Or the human programmers can remind themselves to get scared if they find the AI especially preoccupied with modeling their own thinking.
But as an AI develops, it will become increasingly difficult for its human creators to keep track of everything it thinks, plans, and considers. Perhaps the greatest dangers will be those stratagems which are not explicitly (in the thought process) motivated by the desire to escape human control. The AI simply has a goal, X, and it perceives that ABCDEF... combined will facilitate X, and it all looks innocent to the human supervisors. But actually ABC is a miracle self-hack which lets a plan get developed without being analyzed by module D, and module D has the property E of making planning really slow, which by heuristic F is a bad thing. Unfortunately, module D is part of the automatic system which looks for consideration of dangerous ...
Focusing on slow-developing uploads doesn't cause slower development of other forms of AGI. Uploads themselves can't be expected to turn into FAIs without developing the (same) clean theory of de novo FAI (people are crazy, and uploads are no exception; this is why we have existential risk in the first place, even without any uploads). It's very hard to incrementally improve uploads' intelligence without affecting their preference, and so that won't happen on the first steps from vanilla humans, and pretty much can't happen unless we already have a good th...
The only way in which uploads might help on the way towards FAI is by being faster (or even smarter/saner) FAI theorists, but in this regard they may accelerate the arrival of existential risks as well (especially the faster uploads that are not smarter/saner).
Emulations could also enable the creation of a singleton capable of globally balancing AI development speeds and dangers. That singleton could then take billions of subjective years to work on designing safe and beneficial AI. If designing safe AI is much, much harder than building AI at all, or if knowledge of AI and safe AI are tightly coupled, such a singleton might be the most likely route to a good outcome.
Okay, let's go on the brain-simulation path. Let's start with something simple, like a lobster or a dog... oh wait, what if it transcends and isn't human-friendly. All right, we'll stick to human brains... oh wait, what if our model of neural function is wrong and we create a sociopathic copy that isn't human-friendly. All right, we'll work on human brain regions separately, and absolutely make sure that we have them all right before we do a whole brain... oh wait, what if one of our partial brain models transcends and isn't human-friendly.
And while you, ...
An important fact is that whether your aim is Friendly AI or mind uploading, either way, someone has to do neuroscience. As the author observes,
Such research [FAI] must not only reverse-engineer consciousness, but also human notions of morality.
In FAI strategy as currently conceived, the AI is the neuroscientist. Through a combination of empirical and deductive means, and with its actions bounded by some form of interim Friendliness (so it doesn't kill people or create conscious sim-people along the way), the AI figures out the human decision archite...
Why should an uploaded superintelligence based on a human copy be any innately safer than an artificial superintelligence? Just because humans are usually friendly doesn't mean a human AI would have to be friendly. This is especially true for a superintelligent human AI, which may not even be comparable to its original human template. Even the friendliest human might be angry and abusive when they're having a bad day.
Your idea that a WBE copy would be easier to undergo a relatively more enhanced supervised, safe growth, is basically an assumption. You woul...
WBE safety could benefit from an existing body of knowledge about human behavior and capabilities, and the spaghetti code of the brain could plausibly impose a higher barrier to rapid self-improvement. And institutions exploiting the cheap copyability of brain emulations could greatly help in stabilizing benevolent motivations.
WBE is a tiny region of the space of AI designs that we can imagine as plausible possibilities, and we have less uncertainty about it than about "whatever non-WBE AI technology comes first." Some architectures might be easier to make safe, and others harder, but if you are highly uncertain about non-WBE AI's properties then you need wide confidence intervals.
WBE also has the nice property that it is relatively all-or-nothing. With de novo AI, designers will be tempted to trade off design safety for speed, but for WBE a design that works at all will be relatively close to the desired motivations (there will still be tradeoffs with emulation brain damage, but the effect seems less severe than for de novo AI). Attempts to reduce WBE risk might just involve preparing analysis and institutions to manage WBE upon development, where AI safety would require control of the development process to avoid intrinsically unsafe designs.
These "Whole Brain Emulation" discussions are surreal for me. I think someone needs to put forward the best case they can find that human brain emulations have much of a chance of coming before engineered machine intelligence.
The efforts in that direction I have witnessed so far seem feeble and difficult to take seriously - while the case that engineered machine intelligence will come first seems very powerful to me.
Without such a case, why spend so much time and energy on a discussion of what-if?
Humans misuse power. It doesn't seem to have occurred to you that humans with power frequently become corrupt. So, you want to emulate humans, in order to avoid corruption, when we know that power corrupts humans? Our brain structures have evolved for how many millions of years, all the while natural selection has been favoring those most efficient at obtaining and exploiting power whenever it has provided a reproduction advantage? I think we're better off with something man-made, not something that's optimized to do that! Downvote
Also, there is a poi...
The only downside of this approach I can see is that an upload-triggered Unfriendly singularity may cause more suffering than an Unfriendly AI singularity; sociopaths may be presumed to have more interest in torture of people than a paperclip-optimizing AI would have.
What about those of us who would prefer indefinite human-directed torture to instantaneous cessation of existence? I have no personal plans to explore masochism in that sort of depth, particularly in a context without the generally-accepted safety measures, but it's not the worst thing I ca...
I agree with a lot of your points about the advantages of WBE vs friendly AI. That said, look at the margins. Quite a few people are already working on WBE. Not very many people are working on friendly AI. Taking this into consideration, I think an extra dollar is better spent on FAI research than WBE research.
Also, a world of uploads without FAI would probably not preserve human values for long. The uploads that changed themselves in such a way to grow faster (convert the most resources or make the most copies of themselves) would replace uploads that pre...
Whole Brain Emulation will likely come long after engineered artificial intelligence arrives. Why pump money into Whole Brain Emulation projects? They will still come too late - even with more funding. I figure it's like throwing money down the drain.
It is the fashion in some circles to promote funding for Friendly AI research as a guard against the existential threat of Unfriendly AI. While this is an admirable goal, the path to Whole Brain Emulation is in many respects more straightforward and presents fewer risks. Accordingly, by working towards WBE, we may be able to "weight" the outcome probability space of the singularity such that humanity is more likely to survive.
One of the potential existential risks in a technological singularity is that the recursively self-improving agent might be inimical to our interests, either through actual malevolence or "mere" indifference towards the best interests of humanity. Eliezer has written extensively on how a poorly-designed AI could lead to this existential risk. This is commonly termed Unfriendly AI.
Since the first superintelligence can be presumed to have an advantage over any subsequently-arising intelligences, Eliezer and others advocate funding research into creating Friendly AI. Such research must not only reverse-engineer consciousness, but also human notions of morality. Unfriendly AI could potentially require only sufficiently fast hardware to evolve an intelligence via artificial life, as depicted in Greg Egan's short story "Crystal Nights", or it may be created inadvertently by researchers at the NSA or a similar organization. It may be that creating Friendly AI is significantly harder than creating Unfriendly (or Indifferent) AI, perhaps so much so that we are unlikely to achieve it in time to save human civilization.
Fortunately, there's a short-cut we can take. We already have a great many relatively stable and sane intelligences. We merely need to increase their rate of self-improvement. As far as I can tell, developing mind uploading via WBE is a simpler task than creating Friendly AI. If WBE is fast enough to constitute an augmented intelligence, then our augmented scientists can trigger the singularity by developing more efficient computing devices. An augmented human intelligence may have a slower "take-off" than a purpose-built intelligence, but we can reasonably expect it to be much easier to ensure such a superintelligence is Friendly. In fact, this slower take-off will likely be to our advantage; it may increase our odds of being able to abort an Unfriendly singularity.
WBE may also be able to provide us with useful insights into the nature of consciousness, which will aid Friendly AI research. Even if it doesn't, it gets us most of the practical benefits of Friendly AI (immortality, feasible galactic colonization, etc) and makes it possible to wait longer for the rest of the benefits.
But what if I'm wrong? What if it's just as easy to create an AI we think is Friendly as it is to upload minds into WBE? Even in that case, I think it's best to work on WBE first. Consider the following two worlds: World A creates an AI its best scientists believes is Friendly and, after a best-effort psychiatric evaluation (for whatever good that might do) gives it Internet access. World B uploads 1000 of its best engineers, physicists, psychologists, philosophers, and businessmen (someone's gotta fund the research, right?). World B seems to me to have more survivable failure cases; if some of the uploaded individuals turn out to be sociopaths, the rest of them can stop the "bad" uploads from ruining civilization. It seems exceedingly unlikely that we would select a large enough group of sociopaths that the "good" uploads can't keep the "bad" uploads in check.
Furthermore, the danger of uploading sociopaths (or people who become sociopathic when presented with that power) is also a danger that the average person can easily comprehend, compared to the difficulty of ensuring Friendliness of an AI. I believe that the average person is also more likely to recognize where attempts at safeguarding an upload-triggered singularity may go wrong.
The only downside of this approach I can see is that an upload-triggered Unfriendly singularity may cause more suffering than an Unfriendly AI singularity; sociopaths may be presumed to have more interest in torture of people than a paperclip-optimizing AI would have.
Suppose, however, that everything goes right, the singularity occurs, and life becomes paradise by our standards. Can we predict anything of this future? It's a popular topic in science fiction, so many people certainly enjoy the effort. Depending on how we define a "Friendly singularity", there could be room for a wide range of outcomes.
Perhaps the AI rules wisely and well, and can give us anything we want, "save relevance". Perhaps human culture adapts well to the utopian society, as it seems to have done in the universe of The Culture. Perhaps our uploaded descendants set off to discover the secrets of the universe. I think the best way to ensure a human-centric future is to be the self-improving intelligences, instead of merely catching crumbs from the table of our successors.
In my view, the worst kind of "Friendly" singularity would be one where we discover we've made a weakly godlike entity who believes in benevolent dictatorship; if we must have gods, I want them to be made in our own image, beings who can be reasoned with and who can reason with one another. Best of all, though, is that singularity where we are the motivating forces, where we need not worry if we are being manipulated "in our best interest".
Ultimately, I want the future to have room for our mistakes. For these reasons, we ought to concentrate on achieving WBE and mind uploading first.