You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

bogus comments on Open thread, Oct. 12 - Oct. 18, 2015 - Less Wrong Discussion

5 Post author: MrMind 12 October 2015 06:57AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (250)

You are viewing a single comment's thread. Show more comments above.

Comment author: cousin_it 13 October 2015 10:40:40AM *  6 points [-]

I was just rereading Three Worlds Collide today and noticed that my feelings about the ending have changed over the last few years. It used to be obvious to me that the "status quo" ending was better. Now I feel that the "super happy" ending is better, and it's not just a matter of feelings - it's somehow axiomatically better, based on what I know about decision theory.

Namely, the story says that the super happies are smarter and understand humanity's utility function better, and also that they are moral and wouldn't offer a deal unless it was beneficial according to both utility functions being merged (not just according to their value of happiness). Under these conditions, accepting the deal seems like the right thing to do.

Comment author: bogus 15 October 2015 01:03:37AM *  1 point [-]

I think what the "true" (status-quo) ending proves is that the Super-Happies did not accurately model humanity's utility function at all. If they had, they would have proposed a deal where humanity gets rid of most of its pain, but still keeps some, especially those "grim" things that humans actually like (somewhat counter-intuitively). (And perhaps the Babyeaters' thing would then be understood as one of these "grim" things by humans, as it clearly is for the Babyeaters themselves It's not clear if the Superhappies would be willing to acquire this value, though). This is a deal that humans would indeed accept, since it agrees with their values. I think the true moral of this story is that getting human wants right for something like CEV is a hard problem, and making even small mistakes can have big consequences.