The word "artificial" there makes it look like it means more than it does. And humans are just as made of atoms. Let's try it without that:
The term "friendly intelligence" refers to the production of human-benefiting, non-human-harming actions in intelligences that have advanced to the point of making real-world plans in pursuit of goals.
It's only described in terms of its effects, and then only vaguely. We have no idea what it would actually be. The CEV plan doesn't include what it would actually be, it just includes a technological magic step where it's worked out.
This may be better than nothing, but it's not enough to say it's talking about anything that's actually understood in even the vaguest terms.
For an analogy, what would a gorilla-friendly human-level intelligence be like? How would you reasonably make sure it wasn't harmful to the future of gorillas? (Humans out the box do pretty badly at this.) What steps would the human take to ascertain the CEV of gorillas, assuming tremendous technological resources?
We can't answer the "how can you do this?" questions today. If we could we would be done.
It's true that CEV is an 8-year old, moon-onna-stick wishlist - apparently created without much thought about to how to implement it. C'est la vie.
Suppose you buy the argument that humanity faces both the risk of AI-caused extinction and the opportunity to shape an AI-built utopia. What should we do about that? As Wei Dai asks, "In what direction should we nudge the future, to maximize the chances and impact of a positive intelligence explosion?"
This post serves as a table of contents and an introduction for an ongoing strategic analysis of AI risk and opportunity.
Contents:
Why discuss AI safety strategy?
The main reason to discuss AI safety strategy is, of course, to draw on a wide spectrum of human expertise and processing power to clarify our understanding of the factors at play and the expected value of particular interventions we could invest in: raising awareness of safety concerns, forming a Friendly AI team, differential technological development, investigating AGI confinement methods, and others.
Discussing AI safety strategy is also a challenging exercise in applied rationality. The relevant issues are complex and uncertain, but we need to take advantage of the fact that rationality is faster than science: we can't "try" a bunch of intelligence explosions and see which one works best. We'll have to predict in advance how the future will develop and what we can do about it.
Core readings
Before engaging with this series, I recommend you read at least the following articles:
Example questions
Which strategic questions would we like to answer? Muehlhauser (2011) elaborates on the following questions:
Salamon & Muehlhauser (2013) list several other questions gathered from the participants of a workshop following Singularity Summit 2011, including:
These are the kinds of questions we will be tackling in this series of posts for Less Wrong Discussion, in order to improve our predictions about which direction we can nudge the future to maximize the chances of a positive intelligence explosion.