In his last paragraph, he makes the following controversial statement:
For yet another consequence of understanding that the target ability is qualitatively different is that, since humans have it and apes do not, the information for how to achieve it must be encoded in the relatively tiny number of differences between the DNA of humans and that of chimpanzees.
I remember Eliezer making the same point in a bloggingheads video with Robin Hanson. I believe Hanson's position (although I watched this years ago, and I may be confabulating) was that our intelligence works via the same kludge as other animals and we got better results mostly by developing a better talent for social transmission.
That idea makes a lot of sense to me. I recall reading that chimpanzees have at various times invented more advanced tools (e.g. using spears to hunt bush babies )--but they seem to spread the methods by accident rather than deliberate teaching. New chimpanzee technologies don't seem to persist as they do for humans.
ETA: I looked it up, and I couldn't find a Hanson/Yudkowsky bloggingheads. I'm not sure if it was taken down or if the video was not done through bloggingheads.
I remember Eliezer making the same point in a bloggingheads video with Robin Hanson.
A Hanson/Yudkowsky bloggingheads?!? Methinks you are mistaken.
Folks here should be familiar with most of these arguments. Putting some interesting quotes below:
http://aeon.co/magazine/being-human/david-deutsch-artificial-intelligence/
"Creative blocks: The very laws of physics imply that artificial intelligence must be possible. What's holding us up?"
He also says confusing things about induction being inadequate for creativity which I'm guessing he couldn't support well in this short essay (perhaps he explains better in his books). Not quoting here. His attack on Bayesianism as an explanation for intelligence is valid and interesting, but could be wrong. Given what we know about neural networks, something like this does happen in the brain, and possibly even at a concept level.
His final conclusions are disagreeable. He somehow concludes that the principal bottleneck in AGI research is a philosophical one.
In his last paragraph, he makes the following controversial statement:
This would be false if, for example, the mother controls gene expression while a foetus develops and helps shape the brain. We should be able to answer this question definitively once we can grow human babies completely in vitro. Another problem would be the impact of the cultural environment. A way to answer this question would be to see if our Stone Age ancestors would be classified as AGIs under a reasonable definition