Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

LukeStebbing comments on SIAI - An Examination - Less Wrong

143 Post author: BrandonReinhart 02 May 2011 07:08AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (203)

You are viewing a single comment's thread. Show more comments above.

Comment author: LukeStebbing 05 May 2011 10:37:12PM 3 points [-]

If we take those probabilities as a given, they strongly encourage a strategy that increases the chance that the first seed AI is Friendly.

jsalvatier already had a suggestion along those lines:

I wonder if SIAI could publicly discuss the values part of the AI without discussing the optimization part.

A public Friendly design could draw funding, benefit from technical collaboration, and hopefully end up used in whichever seed AI wins. Unfortunately, you'd have to decouple the F and AI parts, which is impossible.

Comment author: jsalvatier 06 May 2011 04:55:05PM 0 points [-]

Isn't CEV an attempt to separate F and AI parts?

Comment author: wedrifid 06 May 2011 05:06:53PM *  2 points [-]

Isn't CEV an attempt to separate F and AI parts?

It's half of the F. Between the CEV and the AGI is the 'goal stability under recursion' part.

Comment author: LukeStebbing 06 May 2011 04:58:19PM 1 point [-]

It's a good first step.

Comment author: jsalvatier 06 May 2011 05:05:31PM 0 points [-]

I don't understand your impossibility comment, then.

Comment author: LukeStebbing 06 May 2011 05:38:03PM *  3 points [-]

I'm talking about publishing a technical design of Friendliness that's conserved under self-improving optimization without also publishing (in math and code) exactly what is meant by self-improving optimization. CEV is a good first step, but a programmatically reusable solution it is not.

On doing the impossible:

Before you the terrible blank wall stretches up and up and up, unimaginably far out of reach. And there is also the need to solve it, really solve it, not "try your best".

Comment author: jsalvatier 06 May 2011 05:44:50PM 2 points [-]

OK, I understand that much better now. Great point.