Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

mitchell_porter2 comments on Failed Utopia #4-2 - Less Wrong

53 Post author: Eliezer_Yudkowsky 21 January 2009 11:04AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (248)

Sort By: Old

You are viewing a single comment's thread.

Comment author: mitchell_porter2 22 January 2009 03:20:00AM 2 points [-]

This is what I think of as a "mildly unfriendly" outcome. People still end up happy, but before the change, they would not have wanted the outcome. One way for that to happen involves the AI forcibly changing value systems, so that everyone suddenly has an enthusiasm for whatever imperatives it wishes to impose. In this story, as I understand it, there isn't even alteration of values, just a situation constructed to induce the victory of one set of values (everything involved in the quest for a loved one) over another set of values (fidelity to the existing loved one), in a way which violates the protagonist's preferred hierarchy of values.