= 27d567bc2d48d9f40e9ccc20ea370b15)
Jandila comments on Holden's Objection 1: Friendliness is dangerous - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (428)
Not that I'm a proponent of voluntary human extinction, but that's an awfully big conditional.
It's not even strictly true. It's entirely conceivable that FAI will lead to the Sol system being converted into a big block of computronium to run human brain simulations. Even if those simulations have trees and animals in them, I think that still counts as the destruction of nature.
But if FAI is based on CEV, then this will only happen if this is the extrapolated wish of everybody. Assuming existence of people truly (even after extrapolation) valuing Nature in its original form, such computroniums won't be forcefully built.
Nope. CEV that functioned only unanimously wouldn't function at all. The course of the future would go to the majority faction. Honestly, I think CEV is a convoluted, muddy mess of an idea that attempts to solve the hard question of how to teach the AI what we want by replacing it with the harder question of how to teach it what we should want. But that's a different debate.
Why not? I believe that at least one unanimous extrapolated wish exists - for (sentient) life on the planet to continue. If FAI ensured that and left everything else for us to decide, I'd be happy.
Antinatalists exist.
That is not by any means guaranteed to be unanimous. I would be very surprised if at least one person didn't want all sapient life to end, deeply enough for that to persist through extrapolation. I mean, look at all the doomsday cults in the world.
Yes, it is only a hypothesis. Until we actually built an AI with such CEV as utility, we cannot know whether it could function. But at least, running it is uncontroversial by definition.
And I think I'll be more surprised if anyone was found who really and truly had a terminal value for universal death. With some strain, I can imagine someone preferring it conditionally, but certainly not absolutely. The members of doomsday cults, I expect, are either misinformed, insincere, or unhappy about something else (which FAI could fix!).
It's quite controversial. Supposing CEV worked exactly as expected, I still wouldn't want it to be done. Neither do some others in this thread. And I'm sure neither would most humans in the street if you were to ask them (and they seriously though about the question).
CEV doesn't and cannot predict that the extrapolated wishes of everybody will perfectly coincide. Rather, it says it will find the best possible compromise. Of course I would prefer my own values to a compromise! Lacking that, I would prefer a compromise over a smaller group whose members were more similar to myself (such as the group of people actually building the AI).
I might choose CEV over something else because plenty of other things are even worse. But CEV is very very far from the best possible thing, or even the best not-totally-implausible AGI I might expect in my actual future.
Any true believer in a better afterlife qualifies: there are billions of people who at least profess such beliefs, so I expect some of them really believe.
What I proposed in this thread is that CEV would forcibly implement only the (extrapolated) wish(es) of literally everyone. Regarding the rest, it is to minimize its influence, leaving all decisions to people.
No, because they believe in afterlife. They do not wish for universal death. Extrapolating their wish with correct knowledge solves the problem.
Well then, as I and others argue elsewhere in the thread, we anticipate there will be no extrapolated wishes that literally everyone agrees on.
(And that's even without considering some meta formulations of CEV that propose to also take into account the wishes of counterfactual people who might exist in the future, and dead ones who existed in the past.)
Lots of people religiously believe that their god has planned (and prophesied) a specific event of drastic universal change, after which future people will stop suffering in this world, or will stop being born to a life of negative utility (end of the world), or will be rescued from horrible eternal torture (Hell), or which is necessary for the true believers to actually be resurrected or to enter the good afterlife. (Obviously people don't believe all of this at once; these are variant examples.)
Some others believe that life in this world is suffering, negative utility, and ought to be stopped for its own sake (stopping the cycle of rebirth).
I'm very dubious of CEV as a model for Friendly AI. I think it's a bad idea for several reasons. So, not that either.
Also, on topic, recall that, when you extrapolate the volition of crazy people, their volition is not, in particular, more sane. It is more as they would like to be. If you see lizard people, you don't want to see lizard people less. You want sharpened senses to detect them better. Likewise, if you extrapolate a serial killer, you don't get Ghandi. You get an incredibly good serial killer.
I don't see how this is possible. One can be dubious about whether it can be defined in the way it is stated, or whether it can be implemented. But assuming it can, why would it be controversial to fulfill the wish(es) of literally everyone, while affecting everything else the least?
Extrapolating volition includes correcting wrong knowledge and increasing intelligence. So, you do stop seeing lizard people if they don't exist.
Serial killers are more interesting example. But they too don't want everyone to die. Assuming serial killers get full knowledge of their condition and sufficient intelligence for understanding it, what would their volition actually be? I don't know, but I'm sure it's not universal death.
Problems:
Extrapolation is poorly defined, and, to me, seems to go in either one of two directions: either you make people more as they would like to be, which throws any ideas of coherence out the window, or you make people 'better' a long a specific axis, in which case you're no longer directing the question back at humanity in a meaningful sense. Even something as simple as removing wrong beliefs (as you imply) would automatically erase any but the very weakest theological notions. There are a lot of people in the world who would die to stop that from happening. So, yes, controversial.
Coherence, one way or another, is unlikely to exist. Humans want a bunch of different things. Smarter, better-informed humans would still want a bunch of different, conflicting things. Trying to satisfy all of them won't work. Trying to satisfy the majority at the expense of the minorities might get incredibly ugly incredibly fast. I don't have a better solution at this time, but I don't think taking some kind of vote over the sum total of humanity is going to produce any kind of coherent plan of action.
It is not clear that CEV as a model for FAI does either of those things.
Maybe there are better plans that don't involve specifically "sentient" "life" continuing of a "planet", the concepts that could all be broken under sufficient optimization pressure, if they don't happen to be optimal. The simplest ones are "planet" and "life": it doesn't seem like a giant ball of simple elements could be the optimal living arrangement, or biological bodies ("life", if that's what you meant) an optimal living substrate.
I assume FAI, which includes full (super-)human understanding of what is actually meant by "sentient life to continue".
"Planet" is a "planet", even if you should be working on something else, which is what I meant by usual concepts breaking down.
Think of "sentient life continuing on the planet" as a single concept, extrapolatable in various directions as becomes necessary. So, "planet" can be substituted by something else.
But it's the only relevant one, when we're talking about CEV. CEV is only useful if FAI is created, so we can take it for granted.