You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Pfft comments on Open thread, Oct. 12 - Oct. 18, 2015 - Less Wrong Discussion

5 Post author: MrMind 12 October 2015 06:57AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (250)

You are viewing a single comment's thread. Show more comments above.

Comment author: cousin_it 13 October 2015 10:40:40AM *  6 points [-]

I was just rereading Three Worlds Collide today and noticed that my feelings about the ending have changed over the last few years. It used to be obvious to me that the "status quo" ending was better. Now I feel that the "super happy" ending is better, and it's not just a matter of feelings - it's somehow axiomatically better, based on what I know about decision theory.

Namely, the story says that the super happies are smarter and understand humanity's utility function better, and also that they are moral and wouldn't offer a deal unless it was beneficial according to both utility functions being merged (not just according to their value of happiness). Under these conditions, accepting the deal seems like the right thing to do.

Comment author: Pfft 13 October 2015 05:19:25PM 0 points [-]

they are moral and wouldn't offer a deal unless it was beneficial according to both utility functions being merged (not just according to their value of happiness).

I guess whether it is beneficial or not depends on what you compare to? They say,

he obvious starting point upon which to build further negotiations, is to combine and compromise the utility functions of the three species until we mutually satisfice, providing compensation for all changes demanded.

So they are aiming for satisficing rather than maximizing utility: according to all three before-the-change moralities, the post-change state of affairs should be acceptable, but not necessarily optimal. Consider these possibilities:

1) Baby-eaters are modified to no longer eat sentient babies; humans are unchanged; Superhappies like art.

2) Baby-eaters are modified to no longer eat sentient babies; humans are pain-free and eat babies; Superhappies like art.

3) Baby-eaters, humans, and Superhappies are all unchanged.

I think the intention of the author is that, according to pre-change human morality, (1) is the optimal choice, (2) is bad but acceptable, and (3) is unacceptable. The superhappies in the story claim that (2) is the only alternative that is acceptable to all three pre-change moralities. So the super-happy ending is beneficial in the sense that it avoids (3), but it's a "bad" ending because it fails to get (1).

Comment author: cousin_it 13 October 2015 06:12:05PM 0 points [-]

Hmm, I guess I interpreted the super happies proposal differently, as saying that humans get compensation for any downgrade from (1) to (2).