I take some issue with Eliezer naming his function for computing morality "right" and calling the pebble sorters "p-right." Shouldn't Eliezer call his morality function "EY-right"? Humans have many similarities in their morality but it's by no means identical. If anything we should attempt to separate the human morality functions into equivalence classes and then start talking about how to bring all of them into compatibility with each other.
In fact, this relates to something that bugged me about the dust speck/torture post. Suppose we had the choice of torturing a pebble sorter for 50 years versus preventing 3^^^3 pebble sorters from getting a speck of dust in their eyes. Does that change the outcome for anyone? Suppose the choice was between torturing a simple paperclip maximizer or making 3^^^3 paperclip maximizers get a dust speck in one of their sensors? Now the interesting question; would anyone torture a pebble sorter for 50 years to prevent 3^^^3 humans from getting a speck of dust in their eyes? Torture a paperclip maximizer? Torture a human to save paperclip maximizers? I think that we intuitively value the morality of the subjects we're computing maximum utility for based on their moral similarity to ourselves, and further that doing this is right. For instance, I find torture to be pretty abhorrent, to the point that I hope a theoretical metaphysicist would choose to give me and 3^^^3-1 other humans a dust speck instead of torture a single human for any length of time. I would even precommit to feeling bad with at least negative utility (dust_speck_in_the_eye - epsilon) if any rational agent could conceivably choose torture over dust specks. If I had to choose whether to torture a human for 50 years or give 3^^^3 people dust specks I would calculate the total utility with torture as lower than with dust specks because I would project my morality onto those 3^^^3 people, all of whom would then be slightly sadder if I were to choose torture than if I were to choose dust specks. If that is not a rational way to calculate the utility functions of other beings then why shouldn't I choose to torture a human for 50 years to maximize the utility of 3^^^3 paperclip maximizers who might be distracted from perfectly producing paperclips by a dust speck?
I would even precommit to feeling bad with at least negative utility (dustspeckintheeye - epsilon) if any rational agent could conceivably choose torture over dust specks. If I had to choose whether to torture a human for 50 years or give 3^^^3 people dust specks I would calculate the total utility with torture as lower than with dust specks because I would project my morality onto those 3^^^3 people, all of whom would then be slightly sadder if I were to choose torture than if I were to choose dust specks.
If your objective is to make it right to choose...
Today's post, The Bedrock of Morality: Arbitrary? was originally published on 14 August 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Is Fairness Arbitrary?, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.