paulfchristiano comments on Motivating Optimization Processes - Less Wrong

5 Post author: paulfchristiano 22 December 2010 11:36PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (23)
Tags: ai

Comments (23)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 22 December 2010 11:36:05PM 0 points [-]

I personally suspect that it will basically come down to solving the stability under self-modification problem... This may already be the part of the problem that people in the know think is difficult

Yes, it is already the part we suspect will be difficult. The other part may or may not be difficult once we solve that part.

is the above discussion just me starting to suspect how hard friendliness is?

No, you still haven't started to suspect that.

Also, moving this post to the Discussion section.

Comment author: paulfchristiano 23 December 2010 02:23:51AM *  5 points [-]

A long time ago you described what you perceived as the difficulties for FAI:

  1. Solving the technical problems required to maintain a well-specified abstract invariant in a self-modifying goal system. (Interestingly, this problem is relatively straightforward from a theoretical standpoint.)
  2. Choosing something nice to do with the AI. This is about midway in theoretical hairiness between problems 1 and 3.
  3. Designing a framework for an abstract invariant that doesn't automatically wipe out the human species. This is the hard part.

I know that was a long time ago, but people here still link to it, presumably because they don't know of any more up to date statement with similar content. Hopefully you can see why I was confused about which part of this problem was supposed to be hard. I now see that I probably misinterpreted it, but the examples which come directly afterwards reaffirm my incorrect interpretation.

So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don't understand, already requires solving the hard part?

Comment author: Eliezer_Yudkowsky 23 December 2010 03:48:40AM 5 points [-]

Either my views changed since that time, or what I was trying to communicate by it was that 3 was the most inscrutable part of the problem to people who try to tackle it, rather than that it was the blocker problem. 1 is the blocker problem, I think. I probably realize that now to a greater degree than I did at that time, and probably also made more progress on 3 relative to 1, but I don't know how much my opinions actually changed (there's some well-known biases about that).

So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don't understand, already requires solving the hard part?

Current estimate says yes. There would still be an inscrutable problem to solve too, but I don't think it would have quite the same impenetrability about it.

Comment author: paulfchristiano 23 December 2010 05:00:31AM 6 points [-]

Would it be fair to say that even developing a formalism which is capable of precisely expressing the idea that something is a good paperclipper is significantly beyond current techniques, and that substantial progress on this problem probably represents substantial progress towards FAI?

Comment author: Eliezer_Yudkowsky 23 December 2010 05:43:23AM 2 points [-]

Yes.

Powered by Reddit Powered by Reddit