Vladimir_Nesov comments on Poll: What value extra copies? - Less Wrong

5 [deleted] 22 June 2010 12:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (136)

You are viewing a single comment's thread.

Comment author: Vladimir_Nesov 22 June 2010 01:44:31PM 2 points [-]

This question is no good. Would you choose to untranslatable-1 or untranslatable-2? I very much doubt that reliable understanding of this can be reached using human-level philosophy.

Comment author: Nisan 22 June 2010 02:01:34PM 0 points [-]

What kind of philosophy do we need, then?

Comment author: Vladimir_Nesov 22 June 2010 02:44:03PM *  5 points [-]

This is a question about moral estimation. Simple questions of moral estimation can be resolved by observing reactions of people to situations which they evolved to consider: to save vs. to eat a human baby, for example. For more difficult questions involving unusual or complicated situations, or situations involving contradicting moral pressures, we simply don't have any means for extraction of information about their moral value. The only experimental apparatus we have are human reactions, and this apparatus has only so much resolution. Quality of theoretical analysis of observations made using this tool is also rather poor.

To move forward, we need better tools, and better theory. Both could be obtained by improving humans, by making smarter humans that can consider more detailed situations and perform moral reasoning about them. This is not the best option, since we risk creating "improved" humans that have slightly different preferences, and so moral observations obtained using the "improved" humans will be about their preference and not ours. Nonetheless, for some general questions, such as the value of copies, I expect that the answers given by such instruments would also be true about out own preference.

Another way is of course to just create a FAI, which will necessarily be able to do moral estimation of arbitrary situations.

Comment deleted 22 June 2010 04:03:50PM *  [-]
Comment author: Vladimir_Nesov 22 June 2010 05:04:09PM 0 points [-]

It's clear what the situations you talk about are, but these are not the kind of situation your brainware evolved to morally estimate. (This is not the case of a situation too difficult to understand, nor is it a case of a situation involving opposing moral pressures.) The "untranslatable" metaphor was intended to be a step further than you interpreted (which is more clearly explained in my second comment).

Comment deleted 22 June 2010 06:19:10PM [-]
Comment author: Wei_Dai 22 June 2010 11:11:06PM 4 points [-]

Roko, have you seen my post The Moral Status of Independent Identical Copies? There are also some links in the comments of that post to earlier discussions.

Comment author: Vladimir_Nesov 22 June 2010 06:29:17PM *  0 points [-]

Will see. I just have very little hope for progress to be made on this particular dead horse. I offered some ideas about how it could turn out that on human level progress can't in principle be made on this question (and some similar ones).

Comment author: orthonormal 22 June 2010 10:27:12PM *  3 points [-]

Can you call this particular issue a 'dead horse' when it hasn't been a common subject of argument before? (I mean, most of the relevant conversations in human history hadn't gone past the sophomoric question of whether a copy of you is really you.)

If you're going to be pessimistic on the prospect of discussion, I think you'd at very least need a new idiom, like "Don't start beating a stillborn horse".

Comment author: wedrifid 22 June 2010 10:52:41PM 0 points [-]

If you're going to be pessimistic on the prospect of discussion, I think you'd at very least need a new idiom, like "Don't start beating a stillborn horse".

I like the analogy!