I have been mulling around a rough and mostly unformed idea in my head regarding AI-first vs IA-first strategies, but I was loathe to try and put it into words until I saw this post, and noticed that one of the scenarios that I consider highly probable was completely absent.
On the basis that subhuman AGI poses minimal risk to humanity, and that IA increases the level of optimization ability required of an AI to be considered human level or above, it seems that there is a substantial probability that an IA-first strategy could lead to a scenario in which no superhuman AGI can be developed because it is economically infeasible to research that field as opposed to optimizing accelerating returns from IA creation and implementation. Development of AI whether friendly or not would certainly occur at a faster pace, but if IA proves to simply be easier than AI, which given our poor ability estimate the difficulty of both approaches may be true, development in that field would continue to outpace it. It could certainly instigate either a fast or slow takeoff event from our current perspective, but from the perspective of enhanced humans it would be simply an extension of existing trends.
A similar argument could be made in regard to Hanson's WBEM based scenarios, through the implication that given the ability to store a mind to some hardware system, it would be more economically efficient to emulate that mind at a faster pace than to parallel process multiple copies of that mind in the same hardware space, and likewise hardware design would trend toward rapid emulation of single workers rather than multiple instances in order to reduce costs accrued by redundancy and increase gains in efficiency accrued by experience. This would imply that mind enhancement of a few high efficiency minds would occur much earlier and that exceptional numbers of emulated workers would be unlikely to be created, but rather that a few high value workers would occupy a large majority of relevant hardware very soon after the creation of such technology.
An IA field with greater pace than AI does of course present its own problems, and I'm not trying to endorse moving towards an IA-first approach with my ramblings. I suppose I'm simply trying to express the belief that discussion of IA as an alternative to AI rather than an instrument toward AI is rather lacking in this forum and I find myself confused as to why.
"Superhuman AI" as the term is generally used is a fixed reference standard, i.e. your average rationalist computer scientist circa 2013. This particular definition has meaning because if we posit that human beings are able to create an AGI, then a first generation superhuman AGI would be able to understand and modify its own source code, thereby starting the FOOM process. If human beings are not smart enough to write an AGI then this is a moot point. But if we are, then we can be sure that once that self-modifying AGI also reaches human-level ca...
Part of the series AI Risk and Opportunity: A Strategic Analaysis. Previous articles on this topic: Some Thoughts on Singularity Strategies, Intelligence enhancement as existential risk mitigation, Outline of possible Singularity scenarios that are not completely disastrous.
Below are my quickly-sketched thoughts on intelligence amplification and FAI, without much effort put into organization or clarity, and without many references.[1] But first, I briefly review some strategies for increasing the odds of FAI, one of which is to work on intelligence amplification (IA).
Some possible “best current options” for increasing the odds of FAI
Suppose you find yourself in a pre-AGI world,[2] and you’ve been convinced that the status quo world is unstable, and within the next couple centuries we’ll likely[3] settle into one of four stable outcomes: FAI, uFAI, non-AI extinction, or a sufficiently powerful global government which can prevent AGI development[4]. And you totally prefer the FAI option. What should you do to get there?
The IA route
Below are some key considerations about the IA route. I’ve numbered them so they’re easy to refer to later. My discussion assumes MIRI’s basic assumptions, including timelines similar to my own AGI timelines.
Below are my thoughts about all this. These are only my current views: other MIRI personnel (including Eliezer) disagree with some of the points below, and I wouldn’t be surprised to change my mind about some of these things after extended discussion (hopefully in public, on Less Wrong).
I doubt (1) is true. I think IQ 130–170 humans could figure out FAI in 50–150 years if they were trying to solve the right problems, and if FAI development wasn’t in a death race with the strictly easier problem of uFAI. If normal smart humans aren’t capable of building FAI in that timeframe, that’s probably for lack of rationality and philosophical skill, not for lack of IQ. And I’m not confident that rationality and philosophical skill predictably improve with IQ after about IQ 140. It’s a good sign that atheism increases with IQ after IQ 140, but on the other hand I know too many high-IQ people who think that (e.g.) an AI that maximizes K-complexity is a win, and also there’s Stanovich’s research on how IQ and rationality come apart. For these reasons, I’m also not convinced (4) would be a large positive effect on our FAI chances.
Can we train people in rationality and philosophical skill beyond that of say, the 95th percentile Less Wronger? CFAR has plans to find out, but they need to grow a lot first to execute such an ambitious research program.
(2) looks awfully hard, unless we can find a powerful IA technique that also, say, gives you a 10% chance of cancer. Then some EAs devoted to building FAI might just use the technique, and maybe the AI community in general doesn’t.
(5) seems right, though I doubt it’ll be a big enough effect to make a difference for the final outcome.
I think (3) is the dominant consideration here, along with the worry about lacking the philosophical skill (but not IQ) to build FAI at all. At the moment, I (sadly) lean toward the view that slower Earths have a better chance at FAI. (Much of my brain doesn’t know this, though: I remember reading the Summers news with glee, and then remembering that on my current model this was actually bad news for FAI.)
I could say more, but I’ll stop for now and see what comes up in discussion.
My thanks to Justin Shovelain for sending me his old notes on the “IA first” case, and to Wei Dai, Carl Shulman, and Eliezer Yudkowsky for their feedback on this post. ↩
Not counting civilizations that might be simulating our world. This matters, but I won’t analyze that here. ↩
There are other possibilities. For example, there could be a global nuclear war that kills all but about 100,000 people, which could set back social, economic, and technological progress by centuries, thus delaying the crucial point in Earth’s history in which it settles into one of the four stable outcomes. ↩
And perhaps also advanced nanotechnology, intelligence amplification technologies, and whole brain emulation. ↩
Thanks to Carl Shulman for making this point. ↩