= 27d567bc2d48d9f40e9ccc20ea370b15)
thomblake comments on Holden's Objection 1: Friendliness is dangerous - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (428)
There are lots of people who do think that's a good thing, and I don't think those people are trolling or particularly insane. There are entire communities where people have sterilized themselves as part of a mission to end humanity (for the sake of Nature, or whatever).
I think those people do have insufficient knowledge and intelligence. For example, the skoptsy sect, who believed they followed the God's will, were, presumably, factually wrong. And people who want to end humanity for the sake of Nature, want that instrumentally - because they believe that otherwise Nature will be destroyed. Assuming FAI is created, this belief is also probably wrong.
You're right in there being people who would place "all non-intelligent life" before "all people", if there was such a choice. But that does not mean they would choose "non-intelligent life" before "non-intelligent life + people".
That depends a lot on what I understand Nature to be.
If Nature is something incompatible with artificial structuring, then as soon as a superhuman optimizing system structures my environment, Nature has been destroyed... no matter how many trees and flowers and so forth are left.
Personally, I think caring about Nature as something independent of "trees and flowers and so forth" is kind of goofy, but there do seem to be people who care about that sort of thing.
What if particular arrangements of flowers, trees and soforth are complex and interconnected, in ways that can be undone to the net detriment of said flowers, trees and soforth? Thinking here of attempts at scientifically "managing" forest resources in Germany with the goal of making them as accessible and productive as possible. The resulting tree farms were far less resistant to disease, climatic abberation, and so on, and generally not very healthy, because it turns out that illegible, sloppy factor that made forest seem less-conveniently organized for human uses was a non-negligible portion of what allowed them to be so productive and robust in the first place.
No individual tree or flower is all that important, but the arrangement is, and you can easily destroy it without necessarily destroying any particular tree or flower. I'm not sure what to call this, and it's definitely not independent of the trees and flowers and soforth, but it can be destroyed to the concrete and demonstrable detriment of what's left.
That's an interesting question, actually.
I don't know forestry from my elbow, but I used to read a blog by someone who was pretty into saltwater fish tanks. Now, one property of these tanks is that they're really sensitive to a bunch of feedback loops that can most easily be stabilized by approximating a wild reef environment; if you get the lighting or the chemical balance of the water wrong, or if you don't get a well-balanced polyculture of fish and corals and random invertebrates going, the whole system has a tendency to go out of whack and die.
This can be managed to some extent with active modification of the tank, and the health of your tank can be described in terms of how often you need to tweak it. Supposing you get the balance just right, so that you only need to provide the right energy inputs and your tank will live forever: is that Nature? It certainly seems to have the factors that your ersatz German forest lacks, but it's still basically two hundred enclosed gallons of salt water hooked up to an aeration system.
That's something like my objection to CEV-- I currently believe that some fraction of important knowledge is gained by blundering around and (or?) that the universe is very much more complex than any possible theory about it.
This means that you can't fully know what your improved (by what standard?) self is going to be like.
It's the difference between the algorithm and its output, and the local particulars of portions of that output.
I'm not quite sure what you mean to ask by the question. If maintaining a particular arrangement of flowers, trees and so forth significantly helps preserve their health relative to other things I might do, and I value their health, then I ought to maintain that arrangement.
Presumably, because their knowledge and intelligence are not extrapolated enough.
Well, I certainly agree that increasing my knowledge and intelligence might have the effect of changing my beliefs about the world in such a way that I stop valuing certain things that I currently value, and I find it likely that the same is true of everyone else, including the folks who care about Nature.
Not that I'm a proponent of voluntary human extinction, but that's an awfully big conditional.
It's not even strictly true. It's entirely conceivable that FAI will lead to the Sol system being converted into a big block of computronium to run human brain simulations. Even if those simulations have trees and animals in them, I think that still counts as the destruction of nature.
But if FAI is based on CEV, then this will only happen if this is the extrapolated wish of everybody. Assuming existence of people truly (even after extrapolation) valuing Nature in its original form, such computroniums won't be forcefully built.
Nope. CEV that functioned only unanimously wouldn't function at all. The course of the future would go to the majority faction. Honestly, I think CEV is a convoluted, muddy mess of an idea that attempts to solve the hard question of how to teach the AI what we want by replacing it with the harder question of how to teach it what we should want. But that's a different debate.
Why not? I believe that at least one unanimous extrapolated wish exists - for (sentient) life on the planet to continue. If FAI ensured that and left everything else for us to decide, I'd be happy.
Antinatalists exist.
That is not by any means guaranteed to be unanimous. I would be very surprised if at least one person didn't want all sapient life to end, deeply enough for that to persist through extrapolation. I mean, look at all the doomsday cults in the world.
Yes, it is only a hypothesis. Until we actually built an AI with such CEV as utility, we cannot know whether it could function. But at least, running it is uncontroversial by definition.
And I think I'll be more surprised if anyone was found who really and truly had a terminal value for universal death. With some strain, I can imagine someone preferring it conditionally, but certainly not absolutely. The members of doomsday cults, I expect, are either misinformed, insincere, or unhappy about something else (which FAI could fix!).
It's quite controversial. Supposing CEV worked exactly as expected, I still wouldn't want it to be done. Neither do some others in this thread. And I'm sure neither would most humans in the street if you were to ask them (and they seriously though about the question).
CEV doesn't and cannot predict that the extrapolated wishes of everybody will perfectly coincide. Rather, it says it will find the best possible compromise. Of course I would prefer my own values to a compromise! Lacking that, I would prefer a compromise over a smaller group whose members were more similar to myself (such as the group of people actually building the AI).
I might choose CEV over something else because plenty of other things are even worse. But CEV is very very far from the best possible thing, or even the best not-totally-implausible AGI I might expect in my actual future.
Any true believer in a better afterlife qualifies: there are billions of people who at least profess such beliefs, so I expect some of them really believe.
I'm very dubious of CEV as a model for Friendly AI. I think it's a bad idea for several reasons. So, not that either.
Also, on topic, recall that, when you extrapolate the volition of crazy people, their volition is not, in particular, more sane. It is more as they would like to be. If you see lizard people, you don't want to see lizard people less. You want sharpened senses to detect them better. Likewise, if you extrapolate a serial killer, you don't get Ghandi. You get an incredibly good serial killer.
Maybe there are better plans that don't involve specifically "sentient" "life" continuing of a "planet", the concepts that could all be broken under sufficient optimization pressure, if they don't happen to be optimal. The simplest ones are "planet" and "life": it doesn't seem like a giant ball of simple elements could be the optimal living arrangement, or biological bodies ("life", if that's what you meant) an optimal living substrate.
I assume FAI, which includes full (super-)human understanding of what is actually meant by "sentient life to continue".
"Planet" is a "planet", even if you should be working on something else, which is what I meant by usual concepts breaking down.
But it's the only relevant one, when we're talking about CEV. CEV is only useful if FAI is created, so we can take it for granted.