You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

jacobt comments on Yet another safe oracle AI proposal - Less Wrong Discussion

2 Post author: jacobt 26 February 2012 11:45PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (33)

You are viewing a single comment's thread. Show more comments above.

Comment author: jacobt 27 February 2012 04:53:11AM *  0 points [-]

I think we disagree on what a specification is. By specification I mean a verifier: if you had something fitting the specification, you could tell if it did. For example we have a specification for "proof that P != NP" because we have a system in which that proof could be written and verified. Similarly, this system contains a specification for general optimization. You seem to be interpreting specification as knowing how to make the thing.

If you give this optimizer the MU Puzzle (aka 2^n mod 3 = 0) it will never figure it out, even though most children will come to the right answer in minutes.

If you define the problem as "find n such that 2^n mod 3 = 0" then everyone will fail the problem. And I don't see why the optimizer couldn't have some code that monitors its own behavior. Sure it's difficult to write, but the point of this system is to go from a seed AI to a superhuman AI safely. And such a function ("consciousness") would help it solve many of the sample optimization problems without significantly increasing complexity.