The diamond case: Even if I did want a diamond, I simulate that I would feel nervous, alarmed even, if I indicated that I wanted it to bring me one box and I was brought a different box instead.

My brief recapitulation of Yudkowsky’s diamond example (which you can read in full in his CEV document) probably misled you a little bit. I expect that you would find Yudkowsky’s more thorough exposition of “extrapolating volition” somewhat more persuasive. He also warns about the obvious moral hazard involved in mere humans claiming to have extrapolated someone else’s volition out to significant distances – it would be quite proper for you to be alarmed about that!

If creating something that would act according to what one would want if one /were/ more intelligent or more moral or more altruistic, then A) that would only be desirable if one were such a person currently instead of being the current self, or B) that would be a good upgraded-replacement-self to let loose on the universe while oneself ceasing to exist without seeking to have one's own will be done (other than on that matter of self-replacement).

Taken to the extreme this belief would imply that every time you gain some knowledge, improve your logical abilities or are exposed to new memes, you are changed into a different person. I’m sure you don’t believe that – this is where the concept of “distance” comes into play: extrapolating to short distance (as in the diamond example) allows you to feel that the extrapolated version of yourself is still you, but medium or long distance extrapolation might cause you to see the extrapolated self as alien.

It seems to me that whether a given extrapolation of you is still “you” is just a matter of definition. As such it is orthogonal to the question of the choice of CEV as an AI Friendliness proposal. If we accept that an FAI must take as input multiple human value sets in order for it to be safe – I think that Yudkowsky is very persuasive on this point in the sequences – then there has to be a way of getting useful output from those value sets. Since our existing value computations are inconsistent in themselves, let alone with each other the AI has to perform some kind of transformations to cohere a useful signal from this input – this screens off any question of whether we’d be happy to run with our existing values (although I’d certainly choose the extrapolated volition in any case). “Knowing more”, “thinking faster”, “growing up closer together” and so on seem like the optimal transformations for it to perform. Short-distance extrapolations are unlikely to get the job done, therefore medium or long-distance extrapolations are simply necessary, whatever your opinion on the selfhood question.

Eliezer says: “If our extrapolated volitions say we don't want our extrapolated volitions manifested, the system replaces itself with something else we want, or vanishes in a puff of smoke.” A possible cause of such an output might be the selfhood concern that you have raised.

I unfortunately lack time at the moment; rather than write a badly-thought-out response to the complete structure of reasoning considered, I will for the moment write fully-thought-out thoughts on minor parts thereof that my (?) mind/curiosity has seized on.

'As for “taking over the world by proxy”, again SUAM applies.': this sentence stands out, but glancing upwards and downwards does not immediately reveal what SUAM refers to. Ctrl+F and looking at all appearances of the term SUAM on the page does not reveal what SUAM refers to. The first page of Google results for 'SUAM' does not reveal what SUAM refers to.

Hopefully SUAM is a reference to an S* U* A* M* acronym used elsewhere in the article or in a different well-known article, but a suggestion may be helpful that if the first then S* U* A* M* (SUAM) would be convenient in terms of phrase->acronym, and if the second then a reference to the location or else the expanded form of the acronym would be convenient.

The diamond case: Even if I did want a diamond, I simulate that I would feel nervous, alarmed even, if I indicated that I wanted it to bring me one box and I was brought a different box instead. I'm reminded--though this is not directly relevant--of Google searches, where I on occasion look up a rare word I'm unfamiliar with, and instead am given a page of results for a different (more common) word, with a question at the top asking me if I instead want to search for the word I searched for.

For Google, I would be much less frustrated if it always gave me the results I asked for, and maybe asked if I wanted to search for something else. (That way, when I do misspell something, I'm rightfully annoyed at myself and rightfully pleased with the search engine's consistent behaviour.) For the diamond case, I would be happy if it for instance noticed that I wanted the diamond and alerted me to its actual location, giving me a chance to change my official decision.

Otherwise, I would be quite worried about it making other such decisions without my official consent, such as "Hmm, you say you want to learn about these interesting branches of physics, but I can tell that you say that because you anticipate doing so will make you happy, so I'll ignore your request and pump your brain full of drugs instead forever.". Even if in most cases the outcome is acceptable, for something to second-guess your desires at all means there's always the possibility of irrevocably going against your will.

People may worry that a life of getting whatever one wants(/asks for) may not be ideal, but I'm reminded of the immortality/bat argument in that a person who gets whatever that person wants would probably not want to give that up for the sake of the benefits that would arguably come with not having those advantages.

In a more general sense, given that I already possess priorities and want them to be fulfilled (and know how I want to fulfill them), I would appreciate an entity helping me to do so, but would not want an entity to fulfill priorities that I don't hold or try to fulfill them in ways which conflict with my chosen methods of fulfilling them. If creating something that would act according to what one woul want if one /were/ more intelligent or more moral or more altruistic, then A) that would only be desirable if one were such a person currently instead of being the current self, or B) that would be a good upgraded-replacement-self to let loose on the universe while oneself ceasing to exist without seeking to have one's own will be done (other than on that matter of self-replacement).

SUAM = shut up and multiply

I unfortunately lack time at the moment; rather than write a badly-thought-out response to the complete structure of reasoning considered, I will for the moment write fully-thought-out thoughts on minor parts thereof that my (?) mind/curiosity has seized on.

'As for “taking over the world by proxy”, again SUAM applies.': this sentence stands out, but glancing upwards and downwards does not immediately reveal what SUAM refers to. Ctrl+F and looking at all appearances of the term SUAM on the page does not reveal what SUAM refers to. The first page of Google results for 'SUAM' does not reveal what SUAM refers to.

Hopefully SUAM is a reference to an S* U* A* M* acronym used elsewhere in the article or in a different well-known article, but a suggestion may be helpful that if the first then S* U* A* M* (SUAM) would be convenient in terms of phrase->acronym, and if the second then a reference to the location or else the expanded form of the acronym would be convenient.

The diamond case: Even if I did want a diamond, I simulate that I would feel nervous, alarmed even, if I indicated that I wanted it to bring me one box and I was brought a different box instead. I'm reminded--though this is not directly relevant--of Google searches, where I on occasion look up a rare word I'm unfamiliar with, and instead am given a page of results for a different (more common) word, with a question at the top asking me if I instead want to search for the word I searched for.

For Google, I would be much less frustrated if it always gave me the results I asked for, and maybe asked if I wanted to search for something else. (That way, when I do misspell something, I'm rightfully annoyed at myself and rightfully pleased with the search engine's consistent behaviour.) For the diamond case, I would be happy if it for instance noticed that I wanted the diamond and alerted me to its actual location, giving me a chance to change my official decision.

Otherwise, I would be quite worried about it making other such decisions without my official consent, such as "Hmm, you say you want to learn about these interesting branches of physics, but I can tell that you say that because you anticipate doing so will make you happy, so I'll ignore your request and pump your brain full of drugs instead forever.". Even if in most cases the outcome is acceptable, for something to second-guess your desires at all means there's always the possibility of irrevocably going against your will.

People may worry that a life of getting whatever one wants(/asks for) may not be ideal, but I'm reminded of the immortality/bat argument in that a person who gets whatever that person wants would probably not want to give that up for the sake of the benefits that would arguably come with not having those advantages.

In a more general sense, given that I already possess priorities and want them to be fulfilled (and know how I want to fulfill them), I would appreciate an entity helping me to do so, but would not want an entity to fulfill priorities that I don't hold or try to fulfill them in ways which conflict with my chosen methods of fulfilling them. If creating something that would act according to what one woul want if one /were/ more intelligent or more moral or more altruistic, then A) that would only be desirable if one were such a person currently instead of being the current self, or B) that would be a good upgraded-replacement-self to let loose on the universe while oneself ceasing to exist without seeking to have one's own will be done (other than on that matter of self-replacement).

If you think that torture is worse than dust specks, at what step do you not go along with the reasoning?

When I first read Eliezer's post on this subject, I was confused by this transitivity argument. It seems reasonable. But even at that point, I questioned the idea that if all of the steps as you outline them seem individually reasonable, but torture instead of dust specks seems unreasonable, it is "obvious" that I should privilege the former output of my value computation over the latter.

My position now is that in fact, thinking carefully about the steps of gradually increasing pain, there will be at least one that I object to (but it's easy to miss because the step isn't actually written down). There is a degree of pain that I experience that is tolerable. Ouch! That's painful. There is an infinitesimally greater degree of pain (although the precise point at which this occurs, in terms of physical causes, depends on my mood or brain state at that particular time) that is just too much. Curses to this pain! I cannot bear this pain!

This seems like a reasonable candidate for the step at which I stop you and say no, actually I would prefer any number of people to experience the former pain, rather than one having to bear the latter - that difference just barely exceeded my basic tolerance for pain. Of course we are talking about the same subjective level of pain in different people - not necessarily caused by the same severity of physical incident.

This doesn't seem ideal. However, it is more compatible with my value computation than the idea of torturing someone for the sake of 3^^^3 people with dust specks in their eyes.

Presumably, there's going to be some variation with how the people are feeling. Given 3^^^3 people, this will mean that I can pretty much find someone under any given amount of pleasure/pain.

...

It's the same numbers both ways -- just different people. The only way you could decide which is better is if you care more or less than average about Alice.

If Yudkowsky had set up his thought experiment in this way, I would agree with him. But I don't believe there's any reason to expect there to be a distribution of pain in the way that you describe - or in any case it seems like Yudkowsky's point should generalise, and I'm not sure that it does.

If all 3^^^3 + 1 people are on the pain level of 0, and then I have the choice of bringing them all up to pain level 1 or leaving 3^^^3 of them on pain level 0 and bringing one of them up to pain level 1,000,000,000,000 - I would choose the former.

I may have increased the number of pain units in existence, but my value computation doesn't work by adding up "pain units". I'm almost entirely unconcerned about 3^^^3 people experiencing pain level 1; they haven't reached my threshold for caring about the pain they are experiencing. On the other hand, the individual being tortured is way above this threshold and so I do care about him.

I don't know where the threshold(s) are, but I'm sure that if my brain was examined closely there would be some arbitrary points at which it decides that someone else's pain level has become intolerable. Since these jumps are arbitrary, this would seem to break the idea that "pain units" are additive.

('Should it fit in a pocket or backpack?': Robot chassis, please. 'Who is the user?': Hopefully the consciousness itself. O.O)

• In general, make decisions according to the furtherance of your current set of priorities.
• Personally, though I enjoy certain persistant-world games for their content and lasting internal advantages, the impression I've gotten from reading others' accounts of World of Warcraft compared to other games is that it takes up a disproportionate amount of time/effort/money compared to other sources of pleasure.

For that game, the sunk-costs fallacy and the training-to-do-random-things-infinitely phenomenon may help in speculating about why so many sink and continue to sink time into it. I've noticed that people who bite the bullet and quit speak not as though they were dependent and longing to relapse into remembered joy, but rather as though horrified in retrospect at how they let themselves get used to essentially playing to work, that is doing something which in theory they enjoyed yet which in practice was itself a source of considerable stress/boredom/frustration. (Again, I have had no direct experience with the game.)

For cocaine, straightforwardly there's the expectation that it would do bad things to your receptors (as well as your nose lining...) such that you would gain dependency and require it for normality, as with caffeine and nicotine and alcohol. Your priorities would be forcibly changed to a state incompatible with your current priorities, thus it is worth avoiding. If there were a form or similar thing which in fact had no long-term neurological effects, that is one which actually gave you the high without causing any dependency (is that even theoretically possible, though, considering how the brain works? Well, if dropped to the level of most things then instead, say...), it might be worth trying in the same way that music is helpful to cheer oneself up (if it cost less as well?), or perhaps sweet foods would be a better example there.

The standard answer for sex is that it's already part of your system of priorities, and so there's little helping it. Practically speaking, it would probably be far easier if one could just turn off one's interest in that regard and focus one's energy elsewhere--particularly, in terms of the various psychological/physiological health benefits, if one already cannot experience it yet is near-futilely driven to seek it. Again though, there one more wants to turn off 'the impulse to have sex' rather than sex itself, since there are advantages if you want to have sex and do compared to if you want to and can't, and also advantages if you don't want to and don't compared to if you want to and can't.

Hm... returning to the original question wording, if one treats World of Warcraft as a potentially-addictive use of time that may truly or otherwise effectively rewire one's sytem of priorities to the point of interference with one's current priorities, then it is likely reasonable to avoid it for that reason. It's again important to note which priorities are true priorities (such as improvement of the world?) that one wishes to whole-heartedly support, and which are priorities which, when stopped to think about, don't have a particularly reason to value (such as the sex drive issue, which actually doesn't have much going for it compared to other ways of pursuing pleasure).

(Species-wide reproductive advantages are acknowledged.)

No, because "Alice" was not operating by Crocker's Rules.

Note that these people believing this thing to be true does not in fact make it any likelier to be false. We judge it to be less {more likely to be true} than we would for a generic positing by a generic person, down to the point of no suspicion one way or the other, but this positing is not in fact reversed into a positive impression that something is false.

If one takes two otherwise-identical worlds (unlikely, I grant), one in which a large body of people X posit Y for (patently?) fallacious reasons and one in which that large body of people posit the completely-unrelated Z instead, then it seems that a rational (?) individual in both worlds should have roughly the same impression on whether Y is true or false, rather than in the one world believing Y to be very likely to be false.

One may not give the stupid significant credence when they claim it to be day, but one still doesn't believe it any more likely to be night (than one did before).

((As likely noted elsewhere, the bias-acknowledgement situation results in humans being variably treated as more stupid and less stupid depending on local topic of conversation, due to blind spot specificity.))