Alicorn comments on By Which It May Be Judged - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (934)
Assume the subject of reprogramming is an existing human being, otherwise minimally altered by this reprogramming, i.e., we don't do anything that isn't necessary to switch their motivation to paperclips. So unless you do something gratuitiously non-minimal like moving the whole decision-action system out of the range of introspective modeling, or cutting way down on the detail level of introspective modeling, or changing the empathic architecture for modeling hypothetical selves, the new person will experience themselves as having ineffable 'qualia' associated with the motivation to produce paperclips.
The only way to make it seem to them like their motivational quales hadn't changed over time would be to mess with the encoding of their previous memories of motivation, presumably in a structure-destroying way since the stored data and their introspectively exposed surfaces will not be naturally isomorphic. If you carry out the change to paperclip-motivation in the obvious way, cognitive comparisions of the retrieved memories to current thoughts will return 'unequal ineffable quales', and if the memories are visualized in different modalities from current thoughts, 'incomparable ineffable quales'.
Doing-what-leads-to-paperclips will also be a much simpler 'quale', both from the outside perspective looking at the complexity of cognitive data, and in terms of the internal experience of complexity - unless you pack an awful lot of detail into the question of what constitutes a more preferred paperclip. Otherwise, compared to the old days when you thought about justice and fairness, introspection will show that less questioning and uncertainty is involved, and that there are fewer points of variation among the motivational thought-quales being considered.
I suppose you could put in some extra work to make the previous motivations map in cognitively comparable ways along as many joints as possible, and try to edit previous memories without destroying their structure so that they can be visualized in a least common modality with current experiences. But even if you did, memories of the previous quales for rightness-motivation would appear as different in retrospect when compared to current quales for paperclip-motivation as a memory of a 3D greyscale forest landscape vs. a current experience of a 2D red-and-green fractal, even if they're both articulated in the visual sensory modality and your modal workspace allows you to search for, focus on, and compare commonly 'experienced' shapes between them.
This comment expands how you'd go about reprogramming someone in this way with another layer of granularity, which is certainly interesting on its own merits, but it doesn't strongly support your assertion about what it would feel like to be that someone. What makes you think this is how qualia work? Have you been performing sinister experiments in your basement? Do you have magic counterfactual-luminosity-powers?
I think Eliezer is simply suggesting that qualia don't in fact exist in a vacuum. Green feels the way it does partly because it's the color of chlorophyll. In a universe where plants had picked a different color for chlorophyll (melanophyll, say), with everything else (per impossibile) held constant, we would associate an at least slightly different quale with green and with black, because part of how colors feel is that they subtly remind us of the things that are most often colored that way. Similarly, part of how 'goodness' feels is that it imperceptibly reminds us of the extension of good; if that extension were dramatically different, then the feeling would (barring any radical redesigns of how associative thought works) be different too. In a universe where the smallest birds were ten feet tall, thinking about 'birdiness' would involve a different quale for the same reason.
It sounds to me like you don't think the answer had anything to do with the question. But to think that, you'd pretty much have to discard both the functionalist and physicalist theories of mind, and go full dualist/neutral monist; wouldn't you?
I think I'll go with this as my reply - "Well, imagine that you lived in a monist universe - things would pretty much have to work that way, wouldn't they?"
Possibly (this is total speculation) Eliezer is talking about the feeling of one's entire motivational system (or some large part of it), while you're talking about the feeling of some much narrower system that you identify as computing morality; so his conception of a Clippified human wouldn't share your terminal-ish drives to eat tasty food, be near friends, etc., and the qualia that correspond to wanting those things.
The Clippified human categorizes foods into a similar metric of similarity - still believes that fish tastes more like steak than like chocolate - but of course is not motivated to eat except insofar as staying alive helps to make more paperclips. They have taste, but not tastiness. Actually that might make a surprisingly good metaphor for a lot of the difficulty that some people have with comprehending how Clippy can understand your pain and not care - maybe I'll try it on the other end of that Facebook conversation.
The metaphor seems like it could lose most of its effectiveness on people who have never applied the outside view to how taste and tastiness feel from inside - they've never realized that chocolate tastes good because their brain fires "good taste" when it perceives the experience "chocolate taste". The obvious resulting cognitive dissonance (from "tastes bad for others") predictions match my observations, so I suspect this would be common among non-rationalists. If the Facebook conversation you mention is with people who haven't crossed that inferential gap yet, it might prove not that useful.