Eliezer_Yudkowsky comments on Motivating Optimization Processes - Less Wrong

5 Post author: paulfchristiano 22 December 2010 11:36PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (23)
Tags: ai

Comments (23)

You are viewing a single comment's thread. Show more comments above.

Comment author: paulfchristiano 23 December 2010 02:23:51AM *  5 points [-]

A long time ago you described what you perceived as the difficulties for FAI:

  1. Solving the technical problems required to maintain a well-specified abstract invariant in a self-modifying goal system. (Interestingly, this problem is relatively straightforward from a theoretical standpoint.)
  2. Choosing something nice to do with the AI. This is about midway in theoretical hairiness between problems 1 and 3.
  3. Designing a framework for an abstract invariant that doesn't automatically wipe out the human species. This is the hard part.

I know that was a long time ago, but people here still link to it, presumably because they don't know of any more up to date statement with similar content. Hopefully you can see why I was confused about which part of this problem was supposed to be hard. I now see that I probably misinterpreted it, but the examples which come directly afterwards reaffirm my incorrect interpretation.

So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don't understand, already requires solving the hard part?

Comment author: Eliezer_Yudkowsky 23 December 2010 03:48:40AM 5 points [-]

Either my views changed since that time, or what I was trying to communicate by it was that 3 was the most inscrutable part of the problem to people who try to tackle it, rather than that it was the blocker problem. 1 is the blocker problem, I think. I probably realize that now to a greater degree than I did at that time, and probably also made more progress on 3 relative to 1, but I don't know how much my opinions actually changed (there's some well-known biases about that).

So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don't understand, already requires solving the hard part?

Current estimate says yes. There would still be an inscrutable problem to solve too, but I don't think it would have quite the same impenetrability about it.

Comment author: paulfchristiano 23 December 2010 05:00:31AM 6 points [-]

Would it be fair to say that even developing a formalism which is capable of precisely expressing the idea that something is a good paperclipper is significantly beyond current techniques, and that substantial progress on this problem probably represents substantial progress towards FAI?

Comment author: Eliezer_Yudkowsky 23 December 2010 05:43:23AM 2 points [-]

Yes.

Powered by Reddit Powered by Reddit