Luke Muehlhauser:

[Apr. 8, 2012]

Bill, I'm glad you agreed to discuss artificial general intelligence (AGI) with me. I hope our dialogue will be informative to many readers, and to us!

On what do we agree? In separate conversations, Ben Goertzel and Pei Wang agreed with me on the following statements (though I've clarified the wording for our conversation):

Involuntary death is bad, and can be avoided with the right technology.
Humans can be enhanced by merging with technology.
Humans are on a risky course in general, because powerful technologies can destroy us, humans often behave not in their own best interests, and we are unlikely to voluntarily halt technological progress.
AGI is likely this century.
AGI will greatly transform the world. It poses existential and other serious risks, but could also be the best thing that ever happens to us if we do it right.
Careful effort will be required to ensure that AGI results in good things rather than bad things for humanity.

You stated in private communication that you agree with these statements, so we have substantial common ground.

I'd be curious to learn what you think about AGI safety. If you agree that AGI is an existential risk that will arrive this century, and if you value humanity, one might expect you to think it's very important that we accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI before we develop arbitrary superhuman AGI. (This is what Anna Salamon and I recommend in Intelligence Explosion: Evidence and Import.) What are your thoughts on the matter?

And, which questions would you like to raise?

Bill Hibbard:

[Apr. 11, 2012]

Luke, thanks for the invitation to this dialog and thanks for making my minor edits to your six statements. I agree with them, with the usual reservation that they are predictions and hence uncertain.

Before I answer your other questions it would be useful to set a context by reference to an issue you and Anna raised in your excellent paper, Intelligence Explosion: Evidence and Import. In Section 4.1 you wrote...