I agree with almost all of this article. However, the conclusion that the transition won't happen quite suddenly seems to me to be wrong.
Many things seem to go through a technological stage of progress. Music, for example. It became possible to record it tunefully in the 1900's, and by 1960, recording had essentially been mastered to the fidelity the human ear could hear.
It became possible to create electronic sounds to some extent in the 1950's, and this led to the electric guitar, the Hammond organ, and various analogue synthesizers. Then came digital synthesizers, which over a relatively short time displaced the analogues, and led to a point in the 1990's when it became possible to create any sound. Now your phone is powerful enough to do this.
In the 1970's simple digital light detectors existed. These became consumer digital cameras in the late 1990's. Now we have basically reached a level where pixels are no longer an issue, and prices have dropped immensely.
Digital flat screens were science fiction for decades, and comparatively suddenly became possible, then expensively affordable, then cheaper than CRT's.
In each case there's a longish incubation period where nothing much apparently changes for some years. Then there's a rush of progress over little more than a couple of decades, leading to a new status quo where the old technology is completely displaced.
AI is starting to stir. It's had a long period where initial success was replaced by apparent stasis for some time. But now we are seeing real progress again, and I suspect a period of disruptive change caused by AI technologies is not that far off.
To make a prediction here - we will go from having essentially useless AI to human level AI in around a decade or two - just as we have seen with digital cameras, displays, synths etc. The biggest uncertainty in this is which decade it will be. And the machines won't stop at human level - they will drive straight through and keep going over about a 5 year period. And it's only after that has happened that progress may start speeding up because of it.
In each case there's a longish incubation period where nothing much apparently changes for some years. Then there's a rush of progress over little more than a couple of decades, leading to a new status quo where the old technology is completely displaced.
Uh, to me that looks like 4 examples of gradual progress, and 0 examples of explosions (that is, none that are more like fooms than gradual curves).
Also see: History of the Friendly AI concept.
The ancient atomists reasoned their way from first principles to materialism and atomic theory before Socrates began his life's work of making people look stupid in the marketplace of Athens. Why didn't they discover natural selection, too? After all, natural selection follows necessarily from heritability, variation, and selection, and the Greeks had plenty of evidence for all three pieces. Natural selection is obvious once you understand it, but it took us a long time to discover it.
I get the same vibe from intelligence explosion. The hypothesis wasn't stated clearly until 1965, but in hindsight it seems obvious. (Michael Vassar once told me that once he became a physicalist he said "Oh! Intelligence explosion!" Except of course he didn't know the term "intelligence explosion." And he was probably exaggerating.)
Intelligence explosion follows from physicalism and scientific progress and not much else. Since materialists had to believe that human intelligence resulted from the operation of mechanical systems located in the human body, they could have realized that scientists would eventually come to understand these systems so long as scientific progress continued. (Herophilos and Erasistratus were already mapping which nerves and veins did what back in the 4th century B.C.)
And once human intelligence is understood, it can be improved upon, and this improvement in intelligence can be used to improve intelligence even further. And the ancient Greeks certainly had good evidence that there was plenty of room above us when it came to intelligence.
The major hang-up for predicting intelligence explosion may have been the the inability to imagine that this intelligence-engineering could leave the limitations of the human skull and move to a speedier, more dependable and scalable substrate. And that's why Good's paper had to wait until the age of the computer.
</ speculation>