I suspect a period of disruptive change caused by AI technologies is not that far off.
To make a prediction here - we will go from having essentially useless AI to human level AI in around a decade or two - just as we have seen with digital cameras, displays, synths etc. The biggest uncertainty in this is which decade it will be. And the machines won't stop at human level - they will drive straight through and keep going over about a 5 year period. And it's only after that has happened that progress may start speeding up because of it.
It doesn't sound as though we disagree too much. I expect progress on billion year timescales, though it won't be so dramatic after a while. I'm not arguing for low levels of disruption - but I don't think that systematically exaggerating the expected level of disruption is particularly helpful.
Also see: History of the Friendly AI concept.
The ancient atomists reasoned their way from first principles to materialism and atomic theory before Socrates began his life's work of making people look stupid in the marketplace of Athens. Why didn't they discover natural selection, too? After all, natural selection follows necessarily from heritability, variation, and selection, and the Greeks had plenty of evidence for all three pieces. Natural selection is obvious once you understand it, but it took us a long time to discover it.
I get the same vibe from intelligence explosion. The hypothesis wasn't stated clearly until 1965, but in hindsight it seems obvious. (Michael Vassar once told me that once he became a physicalist he said "Oh! Intelligence explosion!" Except of course he didn't know the term "intelligence explosion." And he was probably exaggerating.)
Intelligence explosion follows from physicalism and scientific progress and not much else. Since materialists had to believe that human intelligence resulted from the operation of mechanical systems located in the human body, they could have realized that scientists would eventually come to understand these systems so long as scientific progress continued. (Herophilos and Erasistratus were already mapping which nerves and veins did what back in the 4th century B.C.)
And once human intelligence is understood, it can be improved upon, and this improvement in intelligence can be used to improve intelligence even further. And the ancient Greeks certainly had good evidence that there was plenty of room above us when it came to intelligence.
The major hang-up for predicting intelligence explosion may have been the the inability to imagine that this intelligence-engineering could leave the limitations of the human skull and move to a speedier, more dependable and scalable substrate. And that's why Good's paper had to wait until the age of the computer.
</ speculation>