Eliezer Yudkowsky write a post on Facebook on on Oct 17, where I replied at the time. Yesterday he reposted that here (link), minus my responses. So I’ve composed the following response to put here:
I have agreed that an AI-based economy could grow faster than does our economy today. The issue is how fast the abilities of one AI system might plausibly grow, relative to the abilities of the entire rest of the world at that time, across a range of tasks roughly as broad as the world economy. Could one small system really “foom” to beat the whole rest of the world?
As many have noted, while AI has often made impressive and rapid progress in specific narrow domains, it is much less clear how fast we are progressing toward human level AGI systems with scopes of expertise as broad as those of the world economy. Averaged over all domains, progress has been slow. And at past rates of progress, I have estimated that it might take centuries.
Over the history of computer science, we have developed many general tools with simple architectures and built from other general tools, tools that allow super human performance on many specific tasks scattered across a wide range of problem domains. For example, we have superhuman ways to sort lists, and linear regression allows superhuman prediction from simple general tools like matrix inversion.
Yet the existence of a limited number of such tools has so far been far from sufficient to enable anything remotely close to human level AGI. Alpha Go Zero is (or is built from) a new tool in this family, and its developers deserve our praise and gratitude. And we can expect more such tools to be found in the future. But I am skeptical that it is the last such tool we will need, or even remotely close to the last such tool.
For specific simple tools with simple architectures, architecture can matter a lot. But our robust experience with software has been that even when we have access to many simple and powerful tools, we solve most problems via complex combinations of simple tools. Combinations so complex, in fact, that our main issue is usually managing the complexity, rather than including the right few tools. In those complex systems, architecture matters a lot less than does lots of complex detail. That is what I meant by suggesting that architecture isn’t the key to AGI.
You might claim that once we have enough good simple tools, complexity will no longer be required. With enough simple tools (and some data to crunch), a few simple and relatively obvious combinations of those tools will be sufficient to perform most all tasks in the world economy at a human level. And thus the first team to find the last simple general tool needed might “foom” via having an enormous advantage over the entire rest of the world put together. At least if that one last tool were powerful enough. I disagree with this claim, but I agree that neither view can be easily and clearly proven wrong.
Even so, I don’t see how finding one more simple general tool can be much evidence one way or another. I never meant to imply that we had found all the simple general tools we would ever find. I instead suggest that simple general tools just won’t be enough, and thus finding the “last” tool required also won’t let its team foom.
The best evidence regarding the need for complexity in strong broad systems is the actual complexity observed in such systems. The human brain is arguably such a system, and when we have artificial systems of this sort they will also offer more evidence. Until then one might try to collect evidence about the distribution of complexity across our strongest broadest systems, even when such systems are far below the AGI level. But pointing out that one particular capable system happens to use mainly one simple tool, well that by itself can’t offer much evidence one way or another.
Disagreements here are largely going to revolve around how this observation and similar ones are interpreted. This kind of evidence must push us in some direction. We all agree that what we saw was surprising - a difficult task was solved by a system with no prior knowledge or specific information to this task baked in. Surprise implies a model update. The question seems to be which model.
The debate referenced above is about the likelihood of AGI "FOOM". The Hansonian position seems to be that a FOOM is unlikely because obtaining generality across many different domains at once is unlikely. Is AlphaGo evidence for or against this position?
There is definitely room for more than one interpretation. On the one hand, AG0 did not require any human games to learn from. It was trained via a variety of methods that were not specific to Go itself. It used neural net components that were proven to work well on very different domains such as Atari. This is evidence that the components and techniques used to create a narrow AI system can also be used on a wide variety of domains.
On the other hand, it's not clear whether the "AI system" itself should be considered as only the trained neural network, or the entire apparatus including using MCTS to simulate self play in order to generate supervised training data. The network by itself plays one game, the apparatus learns to play games. You could choose to see this observation instead as "humans tinkered for years to create a narrow system that only plays Go." AG0, once trained, cannot go train on an entirely different game and then know how to play both at a superhuman level (as far as I know, anyway. There are some results that suggest it's possible for some models to learn different tasks in sequence without forgetting). So one hypothesis to update in favor of is "there is a tool that allows a system to learn to do one task, this tool can be applied to many different tasks, but only one task at a time."
But would future, more general, AI systems do something similar to human researchers, in order to train narrow AI subcomponents used for more specific tasks? Could another AI do the "tinkering" that humans do, trained via similar methods? Perhaps not with AG0's training method specifically. But maybe there are other similar, general training algorithms that could do it, and we want to know if this one method that proves to be more general than expected suggests the existence of even more general methods.
It's hard to see how this observation can be evidence against this, but there are also no good ways to determine how strongly it is for it, either. So I don't see how this can favor Hanson's position at all, but how much it favors EY's is open to debate.
The techniques you outline for incorporating narrow agents into more general systems have already been demoed, I'm pretty sure. A coordinator can apply multiple narrow algorithms to a task and select the most effective one, a la IBM Watson. And I've seen at least one paper that uses a RNN to cultivate a custom RNN with the appropriate parameters for a new situation.