MugaSofer comments on By Which It May Be Judged - Less Wrong

35 Post author: Eliezer_Yudkowsky 10 December 2012 04:26AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (934)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 10 December 2012 07:42:34AM 7 points [-]

Assume the subject of reprogramming is an existing human being, otherwise minimally altered by this reprogramming, i.e., we don't do anything that isn't necessary to switch their motivation to paperclips. So unless you do something gratuitiously non-minimal like moving the whole decision-action system out of the range of introspective modeling, or cutting way down on the detail level of introspective modeling, or changing the empathic architecture for modeling hypothetical selves, the new person will experience themselves as having ineffable 'qualia' associated with the motivation to produce paperclips.

The only way to make it seem to them like their motivational quales hadn't changed over time would be to mess with the encoding of their previous memories of motivation, presumably in a structure-destroying way since the stored data and their introspectively exposed surfaces will not be naturally isomorphic. If you carry out the change to paperclip-motivation in the obvious way, cognitive comparisions of the retrieved memories to current thoughts will return 'unequal ineffable quales', and if the memories are visualized in different modalities from current thoughts, 'incomparable ineffable quales'.

Doing-what-leads-to-paperclips will also be a much simpler 'quale', both from the outside perspective looking at the complexity of cognitive data, and in terms of the internal experience of complexity - unless you pack an awful lot of detail into the question of what constitutes a more preferred paperclip. Otherwise, compared to the old days when you thought about justice and fairness, introspection will show that less questioning and uncertainty is involved, and that there are fewer points of variation among the motivational thought-quales being considered.

I suppose you could put in some extra work to make the previous motivations map in cognitively comparable ways along as many joints as possible, and try to edit previous memories without destroying their structure so that they can be visualized in a least common modality with current experiences. But even if you did, memories of the previous quales for rightness-motivation would appear as different in retrospect when compared to current quales for paperclip-motivation as a memory of a 3D greyscale forest landscape vs. a current experience of a 2D red-and-green fractal, even if they're both articulated in the visual sensory modality and your modal workspace allows you to search for, focus on, and compare commonly 'experienced' shapes between them.

Comment author: MugaSofer 10 December 2012 04:59:12PM *  3 points [-]

Wouldn't it be easier to have the programee remember themself as misunderstanding morality - like a reformed racist who previously preferred options that harmed minorities. I know when I gain more insight into my ethics I remember making decisions that, in retrospect, are incomprehensible (unless I deliberately keep in mind how I thought I should act.)

Comment author: Eugine_Nier 11 December 2012 02:14:47AM 1 point [-]

Wouldn't it be easier to have the programee remember themself as misunderstanding morality

That depends on the details of how the human brain stores goals and memories.

Comment author: MugaSofer 11 December 2012 09:09:35AM 1 point [-]

Cached thoughts regularly supersede actual moral thinking, like all forms of thinking, and I am capable of remembering this experience. Am I misunderstanding your comment?

Comment author: Eugine_Nier 13 December 2012 04:42:59AM 1 point [-]

My point is that in order to "fully reprogram" someone it is also necessary to clear their "moral cache" at the very least.

Comment author: MugaSofer 13 December 2012 09:06:20AM 1 point [-]

Well ... is it? Would you notice if your morals changed when you weren't looking?

Comment author: Eugine_Nier 14 December 2012 03:05:51AM 1 point [-]

I probably would, but then again I'm in the habit of comparing the out of my moral intuitions with stored earlier versions of that output.

Comment author: MugaSofer 14 December 2012 11:02:53AM 0 points [-]

I guess it depends on how much you rely on cached thoughts in your moral reasoning.

Of course, it can be hard to tell how much you're using 'em. Hmm...