bogus comments on Open thread, Oct. 12 - Oct. 18, 2015 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (250)
I was just rereading Three Worlds Collide today and noticed that my feelings about the ending have changed over the last few years. It used to be obvious to me that the "status quo" ending was better. Now I feel that the "super happy" ending is better, and it's not just a matter of feelings - it's somehow axiomatically better, based on what I know about decision theory.
Namely, the story says that the super happies are smarter and understand humanity's utility function better, and also that they are moral and wouldn't offer a deal unless it was beneficial according to both utility functions being merged (not just according to their value of happiness). Under these conditions, accepting the deal seems like the right thing to do.
I think what the "true" (status-quo) ending proves is that the Super-Happies did not accurately model humanity's utility function at all. If they had, they would have proposed a deal where humanity gets rid of most of its pain, but still keeps some, especially those "grim" things that humans actually like (somewhat counter-intuitively). (And perhaps the Babyeaters' thing would then be understood as one of these "grim" things by humans, as it clearly is for the Babyeaters themselves It's not clear if the Superhappies would be willing to acquire this value, though). This is a deal that humans would indeed accept, since it agrees with their values. I think the true moral of this story is that getting human wants right for something like CEV is a hard problem, and making even small mistakes can have big consequences.