I have always been rather nervous about the concept of CEV. Particularly frightening to me are the modifiers "coherent" and "extrapolated". The explanations of these terms in this document strike me as quite incoherent and hence I am forced to extrapolate to get any meaning at all from the phrase. (The fact that the document is more than six years old and proclaims it own obsolescence in its first paragraph does not instill confidence). Of course, this posting gives me further cause for concern. It seems I may also be confused about "volition".
First, let me say why "coherent" frightens me. I wish the word were "collective" instead. It is my understanding that the point of specifying that the volition be "coherent" is that we wish to filter out the incoherent bits of mankind's volition. For example, if mankind's volition were that we not build a monolithic, super-powerful AI in the first place, then that would be an incoherent wish which should be ignored. Or, if mankind's volition did not think that the conquest of death was a high priority, that too would be incoherent and ought to be ignored. The incoherent 'philosophy' of folks below the waterline cannot be allowed to trump the volition of the more rational folk above.
The above is a caricature of 'coherence' as presented in the May 2004 document. If someone else can provide a better interpretation, that would be welcome.
Next, let me say why "extrapolated" frightens me. Extrapolation ought to frighten everyone. An AI has no business looking farther into the future than its human creators. An AI has no need to extrapolate. It has no need to look far ahead into the future. Mankind and its volition are traveling into the future along with the AI. If the AI needs to know what mankind wants 1000 years from now, it should just wait for 1000 years and then ask. It will receive a much better and well informed answer than can be achieved by extrapolating.
Once again, I may be objecting here to a caricature, straw-man interpretation of 'extrapolated'. I wish the word were 'expressed'. Can anyone provide me with a better explanation than Eliezer's (2004) as to why the word 'extrapolated' is the appropriate one?
Also, does anyone have a link to the Wei Dai comments regarding "volition"? The OP's hints make this word just as mysterious and frightening to me as the other two.
The above is a caricature of 'coherence' as presented in the May 2004 document. If someone else can provide a better interpretation, that would be welcome.
That doesn't sound like how I interpreted 'coherent'. I assumed it meant a volition the vast majority of humanity agrees with / a measure of how much humanity's volition agrees. If humanity really didn't care about death, then that would be a coherent volition. So something like 'collective' indeed.
As for extrapolation, it's not intended to literally look into the future. I thought the example of the ...
I know Wei Dai has criticized CEV as a construct, I believe offering the alternative of rigorously specifying volition *before* making an AI. I couldn't find these posts/comments via a search, can anyone link me? Thanks.
There may be related top-level posts, but there is a good chance that what I am specifically thinking of was a comment-level conversation between Wei Dai and Vladimir Nesov.
Also feel free to use this thread to criticize CEV and to talk about other possible systems of volition.