Eliezer,

I am rather surprised that you accept all of the claimed achievements of Eurisko and even regard it as "dangerous", despite the fact that no one save the author has ever seen even a fragment of its source code. I firmly believe that we are dealing with a "mechanical Turk."

I am also curious why you believe that meaningful research on Friendly AI is at all possible without prior exposure to a working AGI. To me it seems a bit like trying to invent the ground fault interrupter before having discovered electricity.

Aside from that: If I had been following your writings more carefully, I might already have learned the answer to this, but: just why do you prioritize formalizing Friendly AI over achieving AI in the fist place? You seem to side with humanity over a hypothetical Paperclip Optimizer. Why is that? It seems to me that unaugmented human intelligence is itself an "unfriendly (non-A)I", quite efficient at laying waste to whatever it touches.

There is every reason to believe that if an AGI does not appear before the demise of cheap petroleum, our species is doomed to "go out with a whimper." I for one prefer the "bang" as a matter of principle.

I would gladly accept taking a chance at conversion to paperclips (or some similarly perverse fate at the hands of an unfriendly AGI) when the alternative appears to be the artificial squelching of the human urge to discover and invent, with the inevitable harvest of stagnation and eventually oblivion.

I accept Paperclip Optimization (and other AGI failure modes) as an honorable death, far superior to being eaten away by old age or being killed by fellow humans in a war over dwindling resources. I want to live in interesting times. Bring on the AGI. It seems to me that if any intelligence, regardless of its origin, is capable of wrenching the universe out of our control, it deserves it.

Why is the continued hegemony of Neolithic flesh-bags so precious to you?