Vladimir_Nesov comments on What is Eliezer Yudkowsky's meta-ethical theory? - Less Wrong

33 Post author: lukeprog 29 January 2011 07:58PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (368)

You are viewing a single comment's thread. Show more comments above.

Comment author: Vladimir_Nesov 09 March 2011 11:53:08AM *  0 points [-]

what Eliezer seems to be arguing is that what we really care about is not (a) that whatever states of affairs our brains are wired to send reward signals in response to be realized, but (b) that we experience peace and love and harmony and discovery and so on.

His motivation for thinking this way is a thought experiment - which might become real in the relatively near future - about what would happen if a superintelligent machine could rewire our brains. If what we really care about is (a), then we shouldn't object if the superintelligent machine rewires our brains to send reward signals only when we are sitting in a jar.

I don't see what plausible reasoning process could lead you to infer this unlikely statement (about motivation, given how many detail would need to be just right for the statement to happen to be true).

Also, even if you forbid modifying definition of human brain, things that initiate high-reward signals in our brains (or that we actually classify as "harmony" or "love") are very far from what we care about, just as whatever a calculator actually computes is not the same kind of consideration as the logically correct answer, even if you use a good calculator and aren't allowed sabotage. There are many reasons (and contexts) for reward in human brain to not be treated as indicative of goodness of a situation.

Comment author: lukeprog 09 March 2011 12:44:26PM *  0 points [-]

I don't understand your second paragraph. It sounds like you are agreeing to me, but your tone suggests you think you are disagreeing with me.

Comment author: Vladimir_Nesov 09 March 2011 04:02:49PM 1 point [-]

It was an explanation for why your thought experiment provides a bad motivation: we can just forbid modification of human brains to stop the thought experiment from getting through, but that would still leave a lot of problems, which shows that just this thought experiment is not sufficient motivation.

Comment author: lukeprog 09 March 2011 07:35:02PM 2 points [-]

Sure, the superintelligence thought experiment is not the fully story.

One problem with the suggestion of writing a rule to not alter human brains comes in specifying how the machine is not allowed to alter human brains. I'm skeptical about our ability to specify that rule in a way that does not lead to disastrous consequences. After all, our brains are being modified all the time by the environment, by causes that are on a wide spectrum of 'direct' and 'indirect.'

Other problems with adding such a rule are given here.

Comment author: Vladimir_Nesov 09 March 2011 08:22:03PM 2 points [-]

(I meant that subjective experience that evaluates situations should be specified using unaltered brains, not that brains shouldn't be altered.)

Comment author: lukeprog 09 March 2011 09:18:57PM 0 points [-]

You've got my curiosity. What does this mean? How would you realize that process in the real world?

Comment author: Vladimir_Nesov 10 March 2011 10:55:54AM 1 point [-]

Come on, this tiny detail isn't worth the discussion. Classical solution to wireheading, asking the original and not the one under the influence, referring to you-at-certain-time and not just you-concept that resolves to something unpredicted at any given future time in any given possible world, rigid-designator-in-time.