Intelligence is not dangerous.
Just like evolution does not care about the well-being of humans a sufficiently intelligent process wouldn’t mind turning us into something new, something instrumentally useful.
An artificial general intelligence just needs to resemble evolution, with the addition of being goal-oriented, being able to think ahead, jump fitness gaps and engage in direct experimentation. But it will care as much about the well-being of humans as biological evolution does, it won’t even consider it if humans are not useful in achieving its terminal goals.
Yes, an AI would understand what “benevolence” means to humans and would be able to correct you if you were going to commit an unethical act. But why would it do that if it is not specifically programmed to do so? Would a polar bear with superior intelligence live together peacefully in a group of bonobo? Why would intelligence cause it to care about the well-being of bonobo?
One can come up with various scenarios of how humans might be instrumentally useful for an AI, but once it becomes powerful enough as to not dependent on human help anymore, why would it care at all?
I wouldn’t bet on the possibility that intelligences implies benevolence. Why would wisdom cause humans to have empathy with a cockroach? Some humans might have empathy with a cockroach, but that is more likely a side effect of our general capacity for altruism that most other biological agents do not share. That some humans care about lower animals is not because they were smart enough to prove some game theoretic conjecture about universal cooperation, it is not a result of intelligence but a coincidental preference that is the result of our evolutionary and cultural history.
At what point between unintelligent processes and general intelligence (agency) do you believe that benevolence and compassion does automatically become part of an agent’s preferences?
Many humans tend to have empathy with other beings and things like robots, based on their superficial resemblance with humans. Seldom ethical behavior is a result of high-level cognition, i.e. reasoning about the overall consequences of a lack of empathy. And even those who do arrive at ethical theories by means of deliberate reflection are often troubled once the underlying mechanisms for various qualities are revealed that are supposed to bear moral significance. Which hints at the fragility of universal compassion and the need to find ways how to consolidate it in powerful agents.
XiXiDu wrote: : "...a sufficiently intelligent process wouldn’t mind turning us into something new, something instrumentally useful."
Why do you state this? Is there any evidence or logic to suppose this?
XiXiDu asks: "Would a polar bear with superior intelligence live together peacefully in a group of bonobo?"
My reply is to ask would a dog or cat live peacefully within a group of humans? Admittedly dogs sometimes bite humans but this aggression is due to a lack of intelligence. Dostoevsky reflects, via Raskolnikov in Crime and Pun...
Suppose you buy the argument that humanity faces both the risk of AI-caused extinction and the opportunity to shape an AI-built utopia. What should we do about that? As Wei Dai asks, "In what direction should we nudge the future, to maximize the chances and impact of a positive intelligence explosion?"
This post serves as a table of contents and an introduction for an ongoing strategic analysis of AI risk and opportunity.
Contents:
Why discuss AI safety strategy?
The main reason to discuss AI safety strategy is, of course, to draw on a wide spectrum of human expertise and processing power to clarify our understanding of the factors at play and the expected value of particular interventions we could invest in: raising awareness of safety concerns, forming a Friendly AI team, differential technological development, investigating AGI confinement methods, and others.
Discussing AI safety strategy is also a challenging exercise in applied rationality. The relevant issues are complex and uncertain, but we need to take advantage of the fact that rationality is faster than science: we can't "try" a bunch of intelligence explosions and see which one works best. We'll have to predict in advance how the future will develop and what we can do about it.
Core readings
Before engaging with this series, I recommend you read at least the following articles:
Example questions
Which strategic questions would we like to answer? Muehlhauser (2011) elaborates on the following questions:
Salamon & Muehlhauser (2013) list several other questions gathered from the participants of a workshop following Singularity Summit 2011, including:
These are the kinds of questions we will be tackling in this series of posts for Less Wrong Discussion, in order to improve our predictions about which direction we can nudge the future to maximize the chances of a positive intelligence explosion.