Two questions about CEV that worry me

cousin_it

Taken from some old comments of mine that never did get a satisfactory answer.

1) One of the justifications for CEV was that extrapolating from an American in the 21st century and from Archimedes of Syracuse should give similar results. This seems to assume that change in human values over time is mostly "progress" rather than drift. Do we have any evidence for that, except saying that our modern values are "good" according to themselves, so whatever historical process led to them must have been "progress"?

2) How can anyone sincerely want to build an AI that fulfills anything except their own current, personal volition? If Eliezer wants the the AI to look at humanity and infer its best wishes for the future, why can't he task it with looking at himself and inferring his best idea to fulfill humanity's wishes? Why must this particular thing be spelled out in a document like CEV and not left to the mysterious magic of "intelligence", and what other such things are there?

Taken from some old comments of mine that never did get a satisfactory answer.

What do you believe about these immutable, universal preferences?

Here are some potential problems I see with these theorized builtin preferences, since we don't know what they actually are yet:

They may conflict with our consciously held morals or desires: e.g., they may not include compassion or altruism for anyone we never met face to face. They may even conflict with both our own morals and with Gork's morals, at the same time. In that case, why shouldn't we privilege our conscious desires?
They may not be very interesting: just "want to have food and comfort, sex, social status, children". They wouldn't include many things we consciously want because those thing evolved out of subverted button-pushing or as hyperstimuli - such as scientific research. Why should we choose to discard such values just because they aren't embodied in our hardware?
They may be susceptible to many cognitive traps and dead-ends (e.g. wireheading) that we can only work around using conscious thought and our consciously held values.
They may include values or desires we would consciously prefer to eradicate entirely, such as a drive for fighting or for making war. If you thought that 1) most people in history enjoyed and desired war, and 2) this was due to a feature of their builtin cognitive architecture that said "when in situation X, conquer your neighbors" - would you want to include this in the CEV?
They may include features that optimize inclusive genetic fitness at the expense of the user, e.g. causing emotional suffering as negative feedback.

They may include values or desires we would consciously prefer to eradicate entirely, such as a drive for fighting or for making war. If you thought that 1) most people in history enjoyed and desired war, and 2) this was due to a feature of their builtin cognitive architecture that said "when in situation X, conquer your neighbors" - would you want to include this in the CEV?

CEV is supposed to incorporate not only the things you want (or enjoy), but also the things you want to want (or don't want to enjoy, in this case).

38

Two questions about CEV that worry me

38

38

38

Two questions about CEV that worry me

38

38