You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

bogus comments on Open thread, Oct. 12 - Oct. 18, 2015 - Less Wrong Discussion

5 Post author: MrMind 12 October 2015 06:57AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (250)

You are viewing a single comment's thread. Show more comments above.

Comment author: bogus 15 October 2015 01:03:37AM *  1 point [-]

I think what the "true" (status-quo) ending proves is that the Super-Happies did not accurately model humanity's utility function at all. If they had, they would have proposed a deal where humanity gets rid of most of its pain, but still keeps some, especially those "grim" things that humans actually like (somewhat counter-intuitively). (And perhaps the Babyeaters' thing would then be understood as one of these "grim" things by humans, as it clearly is for the Babyeaters themselves It's not clear if the Superhappies would be willing to acquire this value, though). This is a deal that humans would indeed accept, since it agrees with their values. I think the true moral of this story is that getting human wants right for something like CEV is a hard problem, and making even small mistakes can have big consequences.