Lessons:

1) A situation with AIs whose intelligence is between village idiot and Einstein -- assuming there is a scale to make "between" a non-poetic concept -- is not very likely and probably short-lived if it does occur (unless perhaps it is engineered that way on purpose).

2) Aspects of human cognition -- our particular emotions, our language forms, perhaps even pervasive mental tricks like reasoning by analogy -- may be irrelevant to Optimization Processes in general, making their focus for AI research possibly "voodoo doll" methodology. AI may only deal with such things as part of communicating with humans, though mastering them well enough to participate effectively in human culture may be as difficult as inventing new technologies.

3) Optimization Processes built by Intelligent Designers can develop in ways that those built by evolution cannot because of multiple coordinated changes (this point has been beaten to death by now I think).

4) Sex is interesting.

For once, I have no complaints. I assume the path is being cleared for a discussion of what actually IS required for an optimization process to do what we need it to do (model the world, improve itself, etc), which seems only marginally related to what our brains do. If that's where this is headed, I'm looking forward to it.

Richard: Thanks for the link; that looks like a bunch of O.B. posts glommed together; I don't find it any more precise or convincing than anything here so far. Don't get me wrong, though; like the suggestive material on O.B. it is very interesting. If it simply isn't possible to get more concrete because the ideas are not developed well enough, so be it.

For the record, my nickname is taken from a character in an old Disney animated film, a (male) deer.

Z.M.: interesting discussion. weapons of math destruction is a wickedly clever phrase. Still, I can hope for more than "FAI must optimize something, we know not what. Before we can figure out what to optimize we have to understand Recursive Self Improvement. But we can't talk about that because it's too dangerous."

Nick: Yes, science is about models, as that post says. Formal models. It does not seem unreasonable to hope that some are forthcoming. Surely that is the goal. The post you reference is complaining about people making a distinction between the theoretical possibility of different levels of intelligence without any rational basis. That doesn't seem to be the same thing as merely asking for a little precision in the definitions of "intelligence", "self improvement", and "friendliness".

Carry on with your long winding road of reasoning.

Of particular interest, which I hope you will dwell on: What does "self-improving" in the context of an AI program mean precisely? If there is a utility function involved, exactly what is it?

I also hope you start introducing some formal notation, to make your speculations on these topics less like science fiction.

"I built my network, and it's massively parallel and interconnected and complicated, just like the human brain from which intelligence emerges! Behold, now intelligence shall emerge from this neural network as well!"

Who actually did this? I'm not aware of any such effort, much less it being a trend. Seems to me that the "AI" side of neural networks is almost universally interested in data processing properties of small networks. Larger more complex network experiments are part of neuroscience (naive in most cases but that's a different topic). I don't think anybody in AI or brain research ever thought their network was or would or could be "intelligent" in the broad sense you are implying.

If the secret report comes back "acceptable risk" I suppose it just gets dumped into the warehouse from Raiders of the Lost Ark, but what if it doesn't?

Perhaps such a report was produced during the construction of the SSC?

What if the report is about something not under monolithic control?

Ben, you could be right that my "world is too fuzzy" view is just mind projection, but let me at least explain what I am projecting. The most natural way to get "unlimited" control over matter is a pure reductionist program in which a formal mathematical logic can represent designs and causal relationships with perfect accuracy (perfect to the limits of quantum probabilities). Unfortunately, combinatorial explosion makes that impractical. What we can actually do instead is redescribe collections of matter in new terms. Sometimes these are neatly linked to the underlying physics and we get cool stuff like f=ma but more often the redescriptions are leaky but useful "concepts". The fact that we have to leak accuracy (usually to the point where definitions themselves are basically impossible) to make dealing with the world tractable is what I mean by "the world is too fuzzy to support much intelligent manipulation". In certain special cases we come up with clever ways to bound probabilities and produce technological wonders... but transhumanist fantasies usually make the leap to assume that all things we desire can be tamed in this way. I think this is a wild leap. I realize most futurists see this as unwarranted pessimism and that the default position is that anything imaginable that doesn't provably violate the core laws of physics only awaits something smart enough to build it.

My other reasons for doubting the ultimate capabilities of RSI probably don't need more explanation. My skepticism about the imminence of RSI as a threat (never mind the overall ability of RSI itself) is more based on the ideas that 1) The world is really damn complicated and it will take a really damn complicated computer to make sense of it (the vast human data sorting machinery is well beyond Roadrunner and is not that capable anyway), and 2) there is still no beginning of a credible theory of how to make sense of a really damn complicated world with software.

I agree it is "very dangerous" to put a low probability on any particular threat being an imminent concern. Many such threats exist and we make this very dangerous tentative conclusion every day... from cancer in our own bodies to bioterror to the possibility that our universe is a simulation designed to measure how long it takes us to find the mass of the Higgs, after which we will be shut off.

That is all just an aside though to my main point, which was that if I'm wrong, the only conclusion I can see is that an explicit program to take over the world with a Friendly AI is the only reasonable option.

I approve of such an effort. If my skepticism is correct it will be impossible for decades at least; if I'm wrong I'd rather have an RSI that at least tried to be Friendly. It does seem that the Friendliness bit is more important than the RSI part as the start of such an effort.

Eliezer taught you rationality, so figure it out!

If I understand the research program under discussion, certain ideas are answered "somebody else will". e.g.

Don't build RSI, build AI with limited improvement capabilities (like humans) and use Moore's law to get speedup. "but somebody else will"

Build it so that all it does is access a local store of data (say a cache of the internet) and answer multiple choice questions (or some other limited function). Don't build it to act. "but somebody else will"

etc. every safety suggestion can be met with "somebody else will build an AI that does not have this safety feature".

So: make it Friendly. "but somebody else won't".

This implies: make it Friendly and help it take over the world to a sufficient degree that "somebody else" has no opportunity to build non-Friendly AI.

I think it is hugely unlikely that intelligence of the level being imagined is possible in anything like the near future, and "recursive self improvement" is very likely to be a lot more limited than projected (there's a limit to how much code can be optimized, P!=NP which severely bounds general search optimization, there's only so much you can do with "probably true" priors, and the physical world itself is too fuzzy to support much intellegent manipulation). But I could be wrong.

So, if you guys are planning to take over the world with your Friendly AI, I hope you get it right. I'm surprised there isn't an "Open Friendliness Project" to help answer all the objections and puzzles that commenters on this thread.

If Friendliness has already been solved, I'm reminded of Dr. Strangelove: it does no good to keep it a secret!

If it isn't, is it moral to work on more dangerous aspects (like reflectivity) without Friendliness worked out beforehand?

There are many terms and concepts that don't pay for themselves, though we might not agree on which ones. For example, I think Goedel's Theorem is one of them... its cuteness and abstract splendor doesn't offset the dumbness it invokes in people trying to apply it. "Consciousness" and "Free Will" are two more.

If the point here is to remove future objections to the idea that AI programs can make choices and still be deterministic, I guess that's fair but maybe a bit pedantic.

Personally I provisionally accept the basic deterministic reductionist view that Eliezer has been sketching out. "Provisionally" because our fundamental view of reality and our place in it has gone through many transformations throughout history and it seems unlikely that exactly today is where such revelations end. But since we don't know what might be next we work with what we have even though it is likely to look naive in retrospect from the future.

The viewpoint also serves to make me happy and relatively carefree... doing important things is fun, achieving successes is rewarding, helping people makes you feel good. Obsessive worry and having the weight of the world on one's shoulders is not fun. "Do what's fun" is probably not the intended lesson to young rationalists, but it works for me!

iwdw: you could be right -- perhaps the truly top talented members of the "next generation" are better attracted to AI by wandering internet blog sequences on "rationality" than some actual progress on AI. I am neither a supergenius nor young so I can't say for sure.