The above is a caricature of 'coherence' as presented in the May 2004 document. If someone else can provide a better interpretation, that would be welcome.
That doesn't sound like how I interpreted 'coherent'. I assumed it meant a volition the vast majority of humanity agrees with / a measure of how much humanity's volition agrees. If humanity really didn't care about death, then that would be a coherent volition. So something like 'collective' indeed.
As for extrapolation, it's not intended to literally look into the future. I thought the example of the diamond in the box was fairly enlightening. The human says 'I want box 1', thinking box 1 contains a diamond. The AI knows the diamond is in box 2, and can extrapolate (as humans do) that the human actually wants the diamond and would ask for box 2 if they knew where the diamond was. The smart AI therefore opens box 2, and the human is happy because they have a diamond. A dumb AI would just give the human box 1 "because they asked for it", even if that's what they didn't really want.
When a lot of humans then say "the conquest of death is not a high priority" the AI extrapolates that if we knew more or had basic rationality training we would say conquest of death is a high priority. And therefore goes about solving death.
At least that's how I understood it.
Yes. The problem is, if you look at the biggest disagreements humans have had - slavery, abortion, regional independence, whom to tax, how much the state should help people who can't help themselves, how much clothing women should wear, whether women should work outside the home - none of them can be resolved in this method. Religion, possibly; but only to the extent that a religion's followers care about the end goal of getting into heaven, and not to the extent that they have internalized its values.
I know Wei Dai has criticized CEV as a construct, I believe offering the alternative of rigorously specifying volition *before* making an AI. I couldn't find these posts/comments via a search, can anyone link me? Thanks.
There may be related top-level posts, but there is a good chance that what I am specifically thinking of was a comment-level conversation between Wei Dai and Vladimir Nesov.
Also feel free to use this thread to criticize CEV and to talk about other possible systems of volition.