The enterprise of achieving it artificially — the field of ‘artificial general intelligence’ or AGI — has made no progress whatever during the entire six decades of its existence.

I disagree with this. The development of probabilistic graphical models (incl. bayesian networks and some types of neural networks) was a very important forward advance, I think.

It remained a guess until the 1980s, when I proved it using the quantum theory of computation.

A little bit of arrogance here from Deutsch, but we can let it slide.

This astounding claim split the intellectual world into two camps, one insisting that AGI was none the less impossible, and the other that it was imminent. Both were mistaken. The first, initially predominant, camp cited a plethora of reasons ranging from the supernatural to the incoherent. All shared the basic mistake that they did not understand what computational universality implies about the physical world, and about human brains in particular.

Absolutely true, and the first camp still persists to this day, and is still extremely confused/ignorant about universality. It's a view that is espoused even in 'popular science' books.

Suppose you were somehow to give them a list, as with the temperature-conversion program, of explanations of Dark Matter that would be acceptable outputs of the program. If the program did output one of those explanations later, that would not constitute meeting your requirement to generate new explanations. For none of those explanations would be new: you would already have created them yourself in order to write the specification. So, in this case, and actually in all other cases of programming genuine AGI, only an algorithm with the right functionality would suffice. But writing that algorithm (without first making new discoveries in physics and hiding them in the program) is exactly what you wanted the programmers to do!

I don't follow. You can write a program to generate random hypotheses, and you can write a program to figure out the implications of those hypotheses and whether they fit in with current experimental data, and if they do, to come up with tests of those ideas for future experiments. Now, just generating hypotheses completely randomly may not be a very efficient way, but it would work. That's very different from saying "It's impossible". It's just a question of figuring out how to make it efficient. So what's the problem here?

Nor can it be met by the technique of ‘evolutionary algorithms’: the Turing test cannot itself be automated without first knowing how to write an AGI program, since the ‘judges’ of a program need to have the target ability themselves.

But the Turing test is very different from coming up with an explanation of dark matter. The Turing test is a very specific test of use of language and common sense, which is only defined in relation to human beings (and thus needs human beings to test) whereas an explanation of dark matter does not need human beings to test. Thus making this particular argument moot.

The prevailing misconception is that by assuming that ‘the future will be like the past’, it can ‘derive’ (or ‘extrapolate’ or ‘generalise’) theories from repeated experiences by an alleged process called ‘induction’. But that is impossible.

What else could it possibly be? Information is either encoded into a brain, or predicted based on past experiences. There is no other way to gain information. Deutsch gives the example of dates starting with 19- or 20-. Surely, such information is not encoded into our brains from birth. It must be learned from past experiences. But knowledge of dates isn't the only knowledge we have! We have teachers and parents telling us about these things so that we can learn how they work. This all falls under the umbrella of 'past experiences'. And, indeed, a machine who's only inputs were dates would have a tough time making meaningful inferences about them, no matter how intelligent or creative it was.

But in reality, only a tiny component of thinking is about prediction at all, let alone prediction of our sensory experiences. We think about the world: not just the physical world but also worlds of abstractions such as right and wrong, beauty and ugliness, the infinite and the infinitesimal, causation, fiction, fears, and aspirations — and about thinking itself.

I cannot make head or tail of this.

Anyway, I stopped reading after this point because it was disappointing. I expected an interesting and insightful argument, one to make me actually question my fundamental assumptions, but that's not the case here.

The prevailing misconception is that by assuming that ‘the future will be like the past’, it can ‘derive’ (or ‘extrapolate’ or ‘generalise’) theories from repeated experiences by an alleged process called ‘induction’. But that is impossible. I myself remember, for example, observing on thousands of consecutive occasions that on calendars the first two digits of the year were ‘19’. I never observed a single exception until, one day, they started being ‘20’. Not only was I not surprised...

I think this paragraph illustrates the key failure of Deutsch's stance: he assumes all statistical methods must be fundamentally naive. This is about equivalent to assuming all statistical methods operate on small data sets. Of course, if your entire dataset is a moderately long list of numbers that all begin with the number 19, your statistical method will naively assume that the 19s will continue with high probability. But this restriction on the size and complexity of the data set is completely arbitrary. Humans experience, and learn from, an enormously vast data set containing language, images, sound, sensorimotor feedback, and more; all of it indexed by a time variable that permits correlational analysis (the man's lips moved in a certain way and the word 'kimchi' came out). The human learning process constructs, with some degree of success, complex world theories that describe this vast data set. When the brain perceives a sequence of dates, as Deutsch mentions, it does not analyze the sequence in isolation and create a simple standalone theory to do the prediction; rather it understands that the sequence is embedded in a much larger web of interrelated data, and correctly applies the complex world theory to produce the right prediction. In other words, though both the data set and the chosen hypothesis are large and complex, the operation is essentially Bayesian in character. Human brains certainly assume the future is like the past, but we know that the past is more complex than a simple sequential theory would predict; when the future is genuinely unlike the past, humans run into serious difficulty.

In his last paragraph, he makes the following controversial statement:

For yet another consequence of understanding that the target ability is qualitatively different is that, since humans have it and apes do not, the information for how to achieve it must be encoded in the relatively tiny number of differences between the DNA of humans and that of chimpanzees.

I remember Eliezer making the same point in a bloggingheads video with Robin Hanson. I believe Hanson's position (although I watched this years ago, and I may be confabulating) was that our intelligence works via the same kludge as other animals and we got better results mostly by developing a better talent for social transmission.

That idea makes a lot of sense to me. I recall reading that chimpanzees have at various times invented more advanced tools (e.g. using spears to hunt bush babies )--but they seem to spread the methods by accident rather than deliberate teaching. New chimpanzee technologies don't seem to persist as they do for humans.

ETA: I looked it up, and I couldn't find a Hanson/Yudkowsky bloggingheads. I'm not sure if it was taken down or if the video was not done through bloggingheads.

Deutsch is interesting. He seems very close to the LW camp, and I think he's someone LWers should at least be familiar with. (This article is not as good an introduction as The Beginning of Infinity, I think.)

I suspect, personally, that the conflict between "Popperian conjecture and criticism" and the LW brand of Bayesianism is a paper tiger. See this comment thread in particular.

Deutsch is right that a huge part of artificial general intelligence is the ability to infer explanatory models from experience from the complete (infinite!) set of possible explanations, rather than just fit parameters to a limited set of hardcoded explanatory models (as AI programs today work). But that's what I think people here think (generally under the name Solomonoff induction).

Deutsch claims in the article to have proved that any physical process can in principle be emulated at arbitrarily fine detail by a universal quantum Turing machine. Is this proof widely accepted? I tried to read the paper, but the math is beyond me. I've found relatively little discussion of it elsewhere, and most of it critical.

A way to answer this question would be to see if our Stone Age ancestors would be classified as AGIs under a reasonable definition

How do you propose this should be done? Put sequenced Stone Age genomes in human ova until a statistically significant number survives long enough for cognitive testing? To get the approximate impact of genes we share with Stone Agers or chimps but not both?

I'm convinced the builders of Stonehenge or the Pyramids were so much less "intelligent" than us that if we met them, we'd think of them as intellectually disabled. But I don't think the knowledge whether that's due to genes or culture is worth that kind of experiment.

For your question about the impact of the cultural environment on "intelligence", twin studies make a lot more sense than what I think you're suggesting.