KatjaGrace comments on Superintelligence 12: Malignant failure modes - Less Wrong

7 Post author: KatjaGrace 02 December 2014 02:02AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (50)

You are viewing a single comment's thread.

Comment author: KatjaGrace 02 December 2014 02:27:08AM 2 points [-]

What do you disagree with most in this section?

Comment author: selylindi 04 December 2014 03:45:59AM *  3 points [-]

I feel like there are malignant failure modes beyond the categories mentioned by Bostrom. Perhaps it would be sensible to try to break down the topic systematically. Here's one attempt.

  1. Design by fools: the AI does what you ask, but you asked for something clearly unfriendly.

  2. Perverse instantiation & infrastructure profusion: the AI does what you ask, but what you ask turns out to be most satisfiable in unforeseen destructive ways, such as redirecting most resources to its infrastructure at our expense.

  3. Partial perverse instantiation & mind crime: the AI does what you ask, which includes both friendly behavior and unfriendly behavior, such as badly treating simulations that have moral status in order to figure out how to treat you well.

  4. Partial instantiation: though the total of what you ask seems friendly, some of what you ask is impossible, the AI does the rest, and the result is imbalanced to an unfriendly degree.

  5. Value drift: changes occur to the AIs code such that it does not do what you ask.