A few months ago, Anna Salamon and I began to write an academic overview of intelligence explosion scenarios — something we could hand to people to explain all our major points in one brief article.
We encountered two major problems.
First: The Summit happened, taking all of our time. Then I was made Executive Director, taking all of my time in a more persistent way.
Second: Being thorough and rigorous in an overview of intelligence explosion requires deep knowledge of a huge spectrum of science and philosophy: history of AI progress, history of planning for the future mattering, AI architectures, hardware progress, algorithms progress, massive datasets, neuroscience, factors in the speed of scientific progress, embryo selection, whole brain emulation, properties of digital minds, AI convergent instrumental values, self-improvement dynamics, takeoff scenarios, heuristics and biases, unipolar and multipolar intelligence explosion scenarios, human values and value extrapolation, decision theory, arms races, human dynamics of technological development, technological forecasting, the economics of machine intelligence, anthropics, evolution, AI-boxing, and much more. Because we were trying to write a short article, we kept having to consume and compress an entire field of knowledge into a single paragraph (or even a single sentence!) with the perfect 2-8 citations, which occasionally meant several days of work for a single paragraph. (This is an extreme example, but it's the kind of problem we often encountered, in different degrees.)
So, we've decided to take a different approach and involve the broader community.
We'll be posting short snippets, short pieces of the puzzle, for feedback from the community. Sometimes we'll pose questions, or ask for references about a given topic, or ask for suggested additions to the dialectic we present.
In the end, we hope to collect and remix the best and most essential snippets, incorporate the feedback and additions provided by the community, and write up the final article.
Think of it as a Polymath Project for intelligence explosion analysis. It's collaborative science and philosophy. Members of Less Wrong tend to be smart, and each one has deep knowledge of one or a few fields that we may not have. We hope you'll join us, and contribute your expertise to this project.
I'll keep a table of contents of all the snippets here, as they are published.
Draft #1:
- Introduction
- Types of digital intelligence
- Why designing digital intelligence gets easier over time
- How long before digital intelligence?
- From digital intelligence to intelligence explosion
- [not finished]
- Snippet 1
- ...
Also see:
I agree with the second sentence (the recommendation, in particular of "explosion scenarios").
As for the first, my agreement is partial. One could say "overview of intelligence explosion", but only if one is thinking of "explosion" specifically as a nominalization of the verb "explode". (Example: "Fireworks commonly explode" -> "The explosion of fireworks is a common occurence".) In this case the phrase "intelligence explosion" would be analogous to a phrase like "product development" (and one can certainly speak of an "overview of product development"). This is however a fairly rare (and rather technical-sounding) usage. The more common meaning of "explosion" is "instance of something exploding", as in "I just saw a huge explosion!". With this meaning, "overview of intelligence explosion" is completely wrong, just like "overview of extinction event" would be.
Like you (I suspect), I initially read the phrase with the second meaning, and it sounded awful at first, before the other interpretation occurred to me a second or two later. This suggests sticking to the second sense of "explosion", and writing "overview of intelligence explosion scenarios".
Thanks. I did consider the other meaning. Note that in "product development", a product is being developed, while in "intelligence explosion", intelligence is exploding. That doesn't by itself make the interpretation impossible; "intelligence explosion" could instead be analogous to "product evolution", say. But the distracting interpretation where intelligence is being exploded made the interpretation of "explosion" as a nominalization of "explode" seem even more unnatural, as did the use elsewhe... (read more)