Mitchell, you criticise my statement as being emotional but are you aware your criticism is emotional. Ironic?
I criticise your statements as unrealistic, wrong, or dogmatic. Calling them emotional is just a way of keeping in view your reasons for making them. I have read your site now so I know this is all about bringing hope to the world, creating a self-fulfilling prophecy, and so on. So here are some more general criticisms.
The promise that "scarcity" will "soon" be abolished doesn't offer hope to anyone except people who are emotionally invested in the idea that no-one should have to have a job. Most people are psychologically adapted to the idea of working for a living. Most people are focused on meeting their own needs. And current "post-scarcity" proposals are impractical social vaporware, so the only hope they offer is to daydreamers hoping that they won't have to interrupt their daydream.
Post-scarcity is apparently about getting everything for free. So if you try to live the dream right now, that means that either someone is giving you things for free, or you make yourself a target for people who want free stuff from you. Some people do manage to avoid working for a living, but none of the existing "methods" - like stealing, inheriting, or marrying someone with a job - can serve as the basis for a whole society. Alternatively, promoting post-scarcity now could mean being an early adopter of technologies which will supposedly be part of a future post-scarcity ensemble; 3D printers are popular in this regard. Well, let's just say that such devices are unreliable, limited in their capabilities, tend to contain high-tech components, and are not going to abolish the economy anyway. I don't doubt that big social experiments are going to be performed as the technological base of such devices improves and expands, but thinking that everything will become fabbed is the 2010s equivalent of the 1990s dream that everything will become virtual. A completely fabbed world is like a completely virtual one; it's a thoroughly unworldly vision; doggedly pursuing it in real life is likely to make you a techno-hobo, squatting in a disused garage along with the junk output of a buggy 3D printer whose feedstock you get on the black market, from dealers catering to the delusions of "maker" utopians. A society and an economy with fabs genuinely at its center must be possible, but there would be enormous creative destruction in getting there from here.
And then we have your long-range ideas. I actually think it's possible that a singularity could lead to a radically better world, but only possible, and your prescription to reject "friendly AI" and related ideas in favor of giving AIs "freedom" is just more wishful thinking. Your ideas about intelligence seem to be based on introspection and intuition - I have in mind, not just what you say about the relation between emotion and reason, but your essay on how friendly AI would cripple the artificial intellect. As I pointed out, the basis of artificial intelligence as it is currently envisaged and pursued is the mathematical theory of computation, algorithms, decision-making, and so on. The philosophy of friendly AI is not about having an autonomous intelligence with preexisting impulses which will then be curbed by Asimov laws; it is about designing the AI so its "impulses" are spontaneously in the right directions. But that is all anthropomorphic psychological language. An artificial intelligence can have a goal system, a problem-solving module, and other components which give it a similar behavior to a conscious being that reasons and emotes; but one doesn't need the psychological language at all to describe such an AI. Arguments from human introspection about the consequences of increased intelligence are essentially irrelevant to the discussion of such AIs, and I don't even consider them a reliable guide to the consequences of superintelligence in a conscious being.
Suppose you buy the argument that humanity faces both the risk of AI-caused extinction and the opportunity to shape an AI-built utopia. What should we do about that? As Wei Dai asks, "In what direction should we nudge the future, to maximize the chances and impact of a positive intelligence explosion?"
This post serves as a table of contents and an introduction for an ongoing strategic analysis of AI risk and opportunity.
Contents:
Why discuss AI safety strategy?
The main reason to discuss AI safety strategy is, of course, to draw on a wide spectrum of human expertise and processing power to clarify our understanding of the factors at play and the expected value of particular interventions we could invest in: raising awareness of safety concerns, forming a Friendly AI team, differential technological development, investigating AGI confinement methods, and others.
Discussing AI safety strategy is also a challenging exercise in applied rationality. The relevant issues are complex and uncertain, but we need to take advantage of the fact that rationality is faster than science: we can't "try" a bunch of intelligence explosions and see which one works best. We'll have to predict in advance how the future will develop and what we can do about it.
Core readings
Before engaging with this series, I recommend you read at least the following articles:
Example questions
Which strategic questions would we like to answer? Muehlhauser (2011) elaborates on the following questions:
Salamon & Muehlhauser (2013) list several other questions gathered from the participants of a workshop following Singularity Summit 2011, including:
These are the kinds of questions we will be tackling in this series of posts for Less Wrong Discussion, in order to improve our predictions about which direction we can nudge the future to maximize the chances of a positive intelligence explosion.