In the early 1980s Douglas Lenat wrote EURISKO, a program Eliezer called "[maybe] the most sophisticated self-improving AI ever built". The program reportedly had some high-profile successes in various domains, like becoming world champion at a certain wargame or designing good integrated circuits.
Despite requests Lenat never released the source code. You can download an introductory paper: "Why AM and EURISKO appear to work" [PDF]. Honestly, reading it leaves a programmer still mystified about the internal workings of the AI: for example, what does the main loop look like? Researchers supposedly answered such questions in a more detailed publication, "EURISKO: A program that learns new heuristics and domain concepts." Artificial Intelligence (21): pp. 61-98. I couldn't find that paper available for download anywhere, and being in Russia I found it quite tricky to get a paper version. Maybe you Americans will have better luck with your local library? And to the best of my knowledge no one ever succeeded in (or even seriously tried) confirming Lenat's EURISKO results.
Today in 2009 this state of affairs looks laughable. A 30-year-old pivotal breakthrough in a large and important field... that never even got reproduced. What if it was a gigantic case of Clever Hans? How do you know? You're supposed to be a scientist, little one.
So my proposal to the LessWrong community: let's reimplement EURISKO!
We have some competent programmers here, don't we? We have open source tools and languages that weren't around in 1980. We can build an open source implementation available for all to play. In my book this counts as solid progress in the AI field.
Hell, I'd do it on my own if I had the goddamn paper.
Update: RichardKennaway has put Lenat's detailed papers up online, see the comments.
rhollerith raised some reasonable objections to this response that I'd like to see answered, but I'll try and answer your question without that information:
As as far as concern goes, I think my threshold for concern over your proposition is identical to my threshold for concern over UFAI, as they postulate similar results (UFAI still seems marginally worse due to the chance of destroying intelligent alien life, but I'll write this off as entirely negligible for the current discussion). I'd say 1:10,000 is a reasonable threshold for concern of the vocalized form, "hey, is anyone looking into this?" I'd love to see some more concrete discussion on this.
"Action" in your scenario is complicated by its direct opposition to acceptance of UFAI, so I can only give you some rough constraints. To simplify, I'll assume all risks allow equally effective action to compensate for them, even though this is clearly not the case.
Let R = the scenario you've described, E = the scenario in which UFAI is a credible threat. "R and E" could be described as "damned if we do, damned if we don't", in which case action is basically futile, so I'll consider the case where R and E are disjoint. In that case, action would only be justifiable if p(R) > p(E). My intuition says that such justification is proportional to p(R) - p(E), but I'd prefer more clarity in this step.
So that's a rough answer... if T is my threshold probability for action in the face of existential risk, T (p(R) - p(E)) is my threshold for action on your scenario. If R and E aren't disjoint, it looks something like T (p(R and ~E) - p(E and ~R)).
A fair answer, thanks.
Though I'm not convinced "R and E" necessarily means "damned either way". If I believed E in addition to R, I think what I would do is:
Forget about memetics in either direction as likely to do more harm than good, and concentrate all available resources on developing Friendly AI as reliably and quickly as possible.
However, provably Friendly AI is still not possible with 2009 vintage tools.
So I'd do it in stages, a series of self improving AIs, the early ones with low intelligence and crude Friendliness architecture... (read more)