= 27d567bc2d48d9f40e9ccc20ea370b15)
gRR comments on Holden's Objection 1: Friendliness is dangerous - LessWrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (428)
What I proposed in this thread is that CEV would forcibly implement only the (extrapolated) wish(es) of literally everyone. Regarding the rest, it is to minimize its influence, leaving all decisions to people.
No, because they believe in afterlife. They do not wish for universal death. Extrapolating their wish with correct knowledge solves the problem.
Well then, as I and others argue elsewhere in the thread, we anticipate there will be no extrapolated wishes that literally everyone agrees on.
(And that's even without considering some meta formulations of CEV that propose to also take into account the wishes of counterfactual people who might exist in the future, and dead ones who existed in the past.)
Lots of people religiously believe that their god has planned (and prophesied) a specific event of drastic universal change, after which future people will stop suffering in this world, or will stop being born to a life of negative utility (end of the world), or will be rescued from horrible eternal torture (Hell), or which is necessary for the true believers to actually be resurrected or to enter the good afterlife. (Obviously people don't believe all of this at once; these are variant examples.)
Some others believe that life in this world is suffering, negative utility, and ought to be stopped for its own sake (stopping the cycle of rebirth).
Well, now you know there exist people who believe that there are some universally acceptable wishes. Let's do the Aumann update :)
False beliefs => irrelevant after extrapolation.
False beliefs (rebirth, existence of nirvana state) => irrelevant after extrapolation.
Aumann update works only if I believe you're a perfect Bayesian rationalist. So, no thanks.
Since you aren't giving any valid examples of universally acceptable wishes (I've pointed out people who don't wish for the examples you gave), why do you believe such wishes exist?
Only if you modify these actual people to have their extrapolated beliefs instead of their current ones. Otherwise the false current beliefs will keep on being very relevant to them. Do you want to do that?
Too bad. Let's just agree to disagree then, until the brain scanning technology is sufficiently advanced.
So far, I didn't see a convincing example of a person who truly wished for everyone to die, even in extrapolation.
To them, yes, but not to their CEV.
Or until you provide the evidence that causes you to hold your opinions.
I think it's plausible such people exist. Conversely, if you fine-tune your implementation of "extrapolation" to make their extrapolated values radically different from their current values (and incidentally matching your own current values), that's not what CEV is supposed to be about. But before talking about that, there's a more important point:
So why do you care about their extrapolated values? If you think CEV will extrapolate something that matches your current values but not those of many others; and you don't want to change by force others' actual values to match their extrapolated ones, so they will suffer in the CEV future; then why extrapolate their values at all? Why not just ignore them and extrapolate your own, if you have the first-mover advantage?
Extrapolated values are the true values. Whereas the current values are approximations, sometimes very bad and corrupted approximations.
This does not follow.
What makes you give them such a label as "true"? There is no such thing as a "correct" or "objective" value. Or values are possible in the sense that there can be agents will all possible values, even paperclip-maximizing. The only interesting property of values is who actually holds them. But nobody actually holds your extrapolated values (today).
Current values (and values in general) are not approximations of any other values. All values just are. Why do you call them approximations?
In your CEV future, the extrapolated values are maximized. Conflicting values, like the actual values held today by many or all people, are necessarily not maximized. In proportion to how much this happens, which is positively correlated to the difference between actual and extrapolated values, people who hold the actual values will suffer living in such a world. (If the AI is a singleton they will not even have a hope of a better future.)
Briefly: suffering ~ failing to achieve your values.
They are reflectively consistent in the limit of infinite knowledge and intelligence. This is a very special and interesting property.
But people would change - gaining knowledge and intelligence - and thus would become happier and happier with time. And I think CEV would try to synchronize this with the timing of its optimization process.
Paperclipping is also self-consistent in that limit. That doesn't make me want to include it in the CEV.
Evidence please. There's a long long leap from ordinary gaining knowledge and intelligence through human life, to "the limit of infinite knowledge and intelligence". Moreover we're considering people who currently explicitly value not updating their beliefs in the face of knowledge, and basing their values on faith not evidence. For all I know they'd never approach your limit in the lifetime of the universe, even if it is the limit given infinite time. And meanwhile they'd be very unhappy.
So you're saying it wouldn't modify the world to fit their new evolved values until they actually evolved those values? Then for all we know it would never do anything at all, and the burden of proof is on you to show otherwise. Or it could modify the world to resemble their partially-evolved values, but then it wouldn't be a CEV, just a maximizer of whatever values people happen to already have.