paulfchristiano comments on Motivating Optimization Processes - Less Wrong

5 Post author: paulfchristiano 22 December 2010 11:36PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (23)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 23 December 2010 03:48:40AM 5 points [-]

Either my views changed since that time, or what I was trying to communicate by it was that 3 was the most inscrutable part of the problem to people who try to tackle it, rather than that it was the blocker problem. 1 is the blocker problem, I think. I probably realize that now to a greater degree than I did at that time, and probably also made more progress on 3 relative to 1, but I don't know how much my opinions actually changed (there's some well-known biases about that).

So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don't understand, already requires solving the hard part?

Current estimate says yes. There would still be an inscrutable problem to solve too, but I don't think it would have quite the same impenetrability about it.

Comment author: paulfchristiano 23 December 2010 05:00:31AM 6 points [-]

Would it be fair to say that even developing a formalism which is capable of precisely expressing the idea that something is a good paperclipper is significantly beyond current techniques, and that substantial progress on this problem probably represents substantial progress towards FAI?

Comment author: Eliezer_Yudkowsky 23 December 2010 05:43:23AM 2 points [-]

Yes.