Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

drnickbone comments on Siren worlds and the perils of over-optimised search - Less Wrong

27 Post author: Stuart_Armstrong 07 April 2014 11:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (411)

You are viewing a single comment's thread.

Comment author: drnickbone 22 April 2014 08:22:29AM 2 points [-]

One issue here is that worlds with an "almost-friendly" AI (one whose friendliness was botched in some respect) may end up looking like siren or marketing worlds.

In that case, worlds as bad as sirens will be rather too common in the search space (because AIs with botched friendliness are more likely than AIs with true friendliness) and a satisficing approach won't work.

Comment author: Stuart_Armstrong 25 April 2014 09:43:34AM 1 point [-]

Interesting thought there...