Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Wei_Dai comments on The Urgent Meta-Ethics of Friendly Artificial Intelligence - Less Wrong

45 Post author: lukeprog 01 February 2011 02:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (249)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 02 February 2011 01:41:43AM 3 points [-]

I'm not sure what you're asking. Are you asking how it is possible that Eliezer's "right" doesn't designate anything, or how that implies Eliezer's meta-ethics is wrong?

Comment author: Vladimir_Nesov 02 February 2011 01:44:18AM *  1 point [-]

I'm asking (1) how is it possible that Eliezer's "right" doesn't designate anything, and (2) how could you arrive at such a strong conclusion based on his non-technical writings, since he could just mean something different, or could have insufficient precision in his own idea to determine this property (this is a meta-point possibly subsumed by the first point).

Comment author: Wei_Dai 02 February 2011 03:50:02AM 13 points [-]

how is it possible that Eliezer's "right" doesn't designate anything

Eliezer identifies "right" with "the ideal morality that I would have if I heard all the arguments, to whatever extent such an extrapolation is coherent." It is possible that human morality, when extrapolated, shows no coherence, in which case Eliezer's "right" doesn't designate anything.

how could you arrive at such a strong conclusion based on his non-technical writings, since he could just mean something different, or could have insufficient precision in his own idea to determine this property

Are you saying that Eliezer's general approach might still turn out to be correct, if we substitute better definitions or understandings of "extrapolation" and/or "coherence"? If so, I agree, and I didn't mean to exclude this possibility with my original statement. Should I have made it clearer when I said "I think Eliezer's meta-ethics is wrong" that I meant "based on my understanding of Eliezer's current ideas"?

Comment author: Vladimir_Nesov 02 February 2011 04:28:44AM 3 points [-]

It is possible that human morality, when extrapolated, shows no coherence

For example, I have no idea what this means. I don't know what "extrapolated" means, apart from some vague intuitions, and even what "coherent" means.

Are you saying that Eliezer's general approach might still turn out to be correct, if we substitute better definitions or understandings of "extrapolation" and/or "coherence"?

Better than what? I have no specific adequate candidates, only a direction of research.

Comment author: Wei_Dai 02 February 2011 08:52:38PM *  2 points [-]

It is possible that human morality, when extrapolated, shows no coherence

For example, I have no idea what this means.

Did you read the thread I linked to in my opening comment, where Marcello and I argued in more detail why we think that? Perhaps we can move the discussion there, so you can point out where you disagree with or not understand us?

Comment author: Vladimir_Nesov 02 February 2011 09:38:31PM *  3 points [-]

To respond to that particular argument, which I don't see how substantiates the point that morality according to Eliezer's meta-ethics could be void.

When you're considering what a human mind would conclude upon considering certain new arguments, you're thinking of ways to improve it. A natural heuristic is to add opportunity for reflection, but obviously exposing one to "unbalanced" argument can lead a human mind anywhere. So you suggest a heuristic of looking for areas of "coherence" in conclusions reached upon exploration of different ways of reflecting.

But this "coherence" is also merely a heuristic. What you want is to improve the mind in the right way, not in coherent way, or balanced way. So you let the mind reflect on strategies for exposing itself to more reflection, and then on strategies for reflecting on reflecting on strategies for getting more reflection, and so on, in any way deemed appropriate by the current implementation. There's probably no escaping this unguided stage, for the most right guide available is the agent itself (unfortunately).

What you end up with won't have opportunity to "regret" past mistakes, for every regret is recognition of an error, and any error can be corrected (for the most part). What's wrong with "incoherent" future growth? Does lack of coherence indicate a particular error, something not done right? If it does, that could be corrected. If it doesn't, everything is fine.

(By the way, this argument could potentially place advanced human rationality and human understanding of decision theory and meta-ethics directly on track to a FAI, with the only way of making a FAI using a human (upload) group self-improvement.)

Comment author: Wei_Dai 02 February 2011 10:56:02PM 4 points [-]

I believe that in Eliezer's meta-ethics, both the extrapolation procedure and the coherence property are to be given fixed logical definitions as part of the meta-ethics, and are not just "heuristics" to be freely chosen by the subject being extrapolated. You seem to be describing your own ideas, which are perhaps similar enough to Eliezer's to be said to fall under his general approach, but I don't think can be said to be Eliezer's meta-ethics.

making a FAI using a human (upload) group self-improvement

Seems like a reasonable idea, but again, almost surely not what Eliezer intended.

Comment author: Vladimir_Nesov 02 February 2011 11:03:47PM 1 point [-]

I believe that in Eliezer's meta-ethics, both the extrapolation procedure and the coherence property are to be given fixed logical definitions as part of the meta-ethics, and are not just "heuristics" to be freely chosen by the subject being extrapolated.

Why "part of meta-ethics"? That would make sense as part of FAI design. Surely the details are not to be chosen "freely", but still there's only one criterion for anything, and that's full morality. For any fixed logical definition, any element of any design, there's a question of what could improve it, make the consequences better.

Comment author: Wei_Dai 02 February 2011 11:10:58PM *  3 points [-]

Why "part of meta-ethics"?

I think because Eliezer wanted to ensure a good chance that right_Eliezer and right_random_human turn out to be very similar. If you let each person choose how to extrapolate using their own current ideas, you're almost certainly going to end up with very different extrapolated moralities.

Comment author: Vladimir_Nesov 02 February 2011 11:28:30PM *  2 points [-]

The point is not that they'll be different, but that mistakes will be made, making the result not quite right, or more likely not right at all. So on the early stage, one must be very careful, develop a reliable theory of how to proceed instead of just doing stuff at random, or rather according to current human heuristics.

Extended amount of reflection looks like one least invasive self-improvement technique, something that's expected to make you more reliably right, especially if you're given opportunity to decide how the process is to be set up. This could get us to the next stage, and so on. More invasive heuristics can prove too disruptive, wrong in unexpected and poorly-understood ways, so that one won't be able to expect the right outcome without close oversight from a moral judgment, which we don't have in any technically strong enough form as of yet.

Comment author: Blueberry 02 February 2011 05:00:16AM 1 point [-]

For example, I have no idea what this means. I don't know what "extrapolated" means, apart from some vague intuitions, and even what "coherent" means.

It means, for instance, that segments of the population who have different ideas on controversial moral questions like abortion or capital punishment actually have different moralities and different sets of values, and that we as a species will never agree on what answers are right, regardless of how much debate or discussion or additional information we have. I strongly believe this to be true.

Comment author: Vladimir_Nesov 02 February 2011 10:48:42AM 2 points [-]

Clearly, I know all this stuff, so I meant something else. Like not having more precise understanding (that could also easily collapse this surface philosophizing).

Comment author: Blueberry 02 February 2011 07:21:23PM 1 point [-]

Well, yes, I know you know all this stuff. Are you saying we can't meaningfully discuss it unless we have a precise algorithmic definition of CEV? People's desires and values are not that precise. I suspect we can only discuss it in vague terms until we come up with some sort of iterative procedure that fits our intuition of what CEV should be, at which point we'll have to operationally define CEV as that procedure.