"More generally, intelligence isn't magical: if there's any way we can tell whether a change in an AGI represents a bug or an improvement, then there's an algorithm that an AI can run to do the same."
Except that we don't - can't - do it by pure armchair thought, which is what the recursive self-improvement proposal amounts to.
The approach of testing a new version in a sandbox had occurred to me, and I agree it is a very promising one for many things - but recursive self-improvement isn't among them! Consider, what's the primary capability for which version N+1 is being tested? Why, the ability to create version N+2... which involves testing N+2... which involves creating N+3... etc.
Again, there's enough correlation between ability to perform certain tasks that you don't need an infinite recursion. To test AIv(N+1)'s ability to program to exact specification, instead of having it program AIv(N+2) have it instead program some other things that AIvN finds difficult (but whose solutions are within AIvN's power to verify). That we will be applying AIv(N+1)'s precision programming to itself doesn't mean we can't test it on non-recursive data first.
ETA: Of course, since we want the end result to be a superintelligence, AIvN might also ask...
In the early 1980s Douglas Lenat wrote EURISKO, a program Eliezer called "[maybe] the most sophisticated self-improving AI ever built". The program reportedly had some high-profile successes in various domains, like becoming world champion at a certain wargame or designing good integrated circuits.
Despite requests Lenat never released the source code. You can download an introductory paper: "Why AM and EURISKO appear to work" [PDF]. Honestly, reading it leaves a programmer still mystified about the internal workings of the AI: for example, what does the main loop look like? Researchers supposedly answered such questions in a more detailed publication, "EURISKO: A program that learns new heuristics and domain concepts." Artificial Intelligence (21): pp. 61-98. I couldn't find that paper available for download anywhere, and being in Russia I found it quite tricky to get a paper version. Maybe you Americans will have better luck with your local library? And to the best of my knowledge no one ever succeeded in (or even seriously tried) confirming Lenat's EURISKO results.
Today in 2009 this state of affairs looks laughable. A 30-year-old pivotal breakthrough in a large and important field... that never even got reproduced. What if it was a gigantic case of Clever Hans? How do you know? You're supposed to be a scientist, little one.
So my proposal to the LessWrong community: let's reimplement EURISKO!
We have some competent programmers here, don't we? We have open source tools and languages that weren't around in 1980. We can build an open source implementation available for all to play. In my book this counts as solid progress in the AI field.
Hell, I'd do it on my own if I had the goddamn paper.
Update: RichardKennaway has put Lenat's detailed papers up online, see the comments.