A summary of the Hanson-Yudkowsky FOOM debate

23 Post author: Kaj_Sotala 15 November 2012 07:25AM

In late spring this year, Luke tasked me with writing a summary and analysis of the Hanson-Yudkowsky FOOM debate, with the intention of having it eventually published in somewhere. Due to other priorities, this project was put on hold for the time being. Because it doesn't look like it will be finished in the near future, and because Curiouskid asked to see it, we thought that we might as well share the thing.

I have reorganized the debate, presenting it by topic rather than in chronological order: I start by providing some brief conceptual background that's useful for understanding Eliezer's optimization power argument, after which I present his argument. Robin's various objections follow, after which there is a summary of Robin's view of how the Singularity will be like, together with Eliezer's objections to that view. Hopefully, this should make the debate easier to follow. This summary also incorporates material from the 90-minute live debate on the topic that they had in 2011. The full table of contents:

  1. Introduction
  2. Overview
  3. The optimization power argument
    1. Conceptual background
    2. The argument: Yudkowsky
    3. Recursive self-improvement
    4. Hard takeoff
    5. Questioning optimization power: the question of abstractions
    6. Questioning optimization power: the historical record
    7. Questioning optimization power: the UberTool question
  4. Hanson's Singularity scenario
    1. Architecture vs. content, sharing of information
    2. Modularity of knowledge
    3. Local or global singularity?
  5. Wrap-up
  6. Conclusions
  7. References

Here's the link to the current draft, any feedback is welcomed. Feel free to comment if you know of useful references, if you think I've misinterpreted something that was said, or if you think there's any other problem. I'd also be curious to hear to what extent people think that this outline is easier to follow than the original debate, or whether it's just as confusing.

Comments (9)

Comment author: niceguyanon 15 November 2012 03:01:59PM 6 points [-]

As someone who has not yet read through all the sequences and found it difficult on a few occasions where I attempted to follow the Hanson-Yudkowsky FOOM debate, I find this summary very helpful.

Comment author: mytyde 20 November 2012 08:39:31AM 3 points [-]

I find it really helps if you do the voices. Like pretend Yudkowsky sounds like Harry Potter from the movie and is having a conversation with Dumbledore (in the HPMOR universe, obviously).

Comment author: bsterrett 16 November 2012 06:47:09PM 3 points [-]

For anyone following the sequence rerun going on right now, this summary is highly recommended. It is much more manageable than the blog posts, and doesn't leave out anything important (that I noticed).

Comment author: mytyde 20 November 2012 08:38:11AM 1 point [-]

...it all makes sense now.

The bleak grey world turns lucid at once!

Comment author: ciphergoth 23 November 2012 10:25:11PM 1 point [-]

This is very useful indeed - thank you!

Comment author: Curiouskid 24 November 2012 01:37:04AM 1 point [-]

Thank you for making this!

I tried to organize it a bit further by porting it to workflowy and grouping some of the bullets together.

Let me know if you find this summary of a summary useful.

Comment author: somervta 19 January 2013 12:47:07AM 1 point [-]

In Conceptual background, bullet 2, you should emphasize the importance of the number of goal states compared to the total number of states - e.g This should also improve bullets 3 & 4.

Comment author: Kaj_Sotala 20 January 2013 01:11:13PM 0 points [-]

Thanks, I implemented that as what is now bullet 3. Do you want to be credited in the acknowledgements, and if so, under what name?

Comment author: somervta 21 January 2013 12:12:37AM 0 points [-]

Sure. Tarn Somervell - more comments may follow when I get around to reading it through.