= 27d567bc2d48d9f40e9ccc20ea370b15)
DanArmak comments on Holden's Objection 1: Friendliness is dangerous - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (428)
Why do you think this is "very likely"?
Today there are many people in the world (gross estimate: tens of percents of world population) who don't believe in noninterference. True believers of several major faiths (most Christian sects, mainstream Islam) desire enforced religious conversion of others, either as a commandment of their faith (for its own sake) or for the metaphysical benefit of those others (to save them from hell). Many people "believe" (if that is the right word) in the subjugation of certain minorities, or of women, children, etc. which involves interference of various kinds. Many people experience future shock which prompts them to want laws that would stop others from self-modifying in certain ways (some including transhumanism).
Why do you think it very likely these people's CEV will contradict their current values and beliefs? Please consider that:
We emphatically don't know the outcome of CEV. If we were sure that it would have any property X, we could hardcode X into the algorithm and make the CEV's task that much easier. Anything you think is very likely for CEV to decide, you should be proportionally willing for me to hardcode into my algorithm, constraining the possible results of CEV.
In these examples, you expect other people's extrapolated values to come to match your actual values. This seems on the outside view like a human bias. Do you expect an equal amount of your important, present-day values to be contradicted and disallowed by humanity's CEV? Can you think of probable examples?
I agree completely - doing the CEV of a small trusted team, who moreover are likely to hold non-extrapolated views similar to ours (e.g. they won't be radical Islamists), would be much better than CEV<humanity>; much more reliable and safe.
But you contradict yourself a little. If you really believed CEV<team> looked a lot like CEV<humanity>, you would have no reason to consider it safer. If you (correctly) think it's safer, that must be because you fear CEV<humanity> will contain some pretty repugnant conclusions that CEV<team> won't.
From this I understand that while you think CEV<team> would have a term for "respecting" the rest of humanity, that respect would be a lot weaker than the equal (and possibly majority-voting-based) rights granted them by CEV<humanity>.
I doubt any one human's values are reflectively consistent. At the very least, every human's values contradict one another in the sense that they compete among themselves for the human's resources, and the human in different moods and at different points in time prefers to spend on different values.
Because infectious memeplexes scare me too, I don't want anyone to build CEV<humanity> (or rather, to run a singleton AI that would implement it) - I would much prefer CEV<small trusted team of people like me> or better CEV<myself> or better yet, a non-CEV process whcih more directly relies on my and other people's non-extrapolated preferences.
A possibly related question: suppose you were about to go off on an expedition in a spaceship that would take you away from Earth for thirty years, and the ship is being stocked with food. Suppose further that, because of an insane bureaucratic process, you have only two choices: either (a) you get to choose what food to stock right now, with no time for nutritional research, or (b) food is stocked according to an expert analysis of your body's nutritional needs, with no input from you. What outcome would you anticipate from each of those choices?
Suppose a hundred arbitrarily selected people were also being sent on similar missions on similar spaceships, and your decision of A or B applied to them as well (either they get to choose their food, or an expert chooses food for them). What outcome would you anticipate from each choice?
I think you meant to add that the expert really understands nutrition, beyond the knowledge of our best nutrition specialists today, which is unreliable and contradictory and sparse.
With that assumption I would choose to rely on the expert, and would expect much less nutritional problems on average for other people who relied on the expert vs. choosing themselves.
The difference between this and CEV is that "what nutritional/metabolic/physiological outcome is good for you" is an objective, pretty well constrained question. There are individual preferences - in enjoyment of food, and in the resulting body-state - but among people hypothetically fully understanding the human body, there will be relatively little disagreement, and the great majority should not suffer much from good choices that don't quite match their personal preferences.
CEV, on the other hand, includes both preferences about objective matters like the above but also many entirely or mostly subjective choices (in the same way that most choices of value are a-rational). Also, people are likely to agree to not interfere in what others eat because they don't often care about it, but people do care about many other behaviors of others (like torturing simulated intelligences, or giving false testimony, or making counterfeit money) and that would be reflected in CEV.
ETA: so in response to your question, I agree that on many subjects I trust experts / CEV more than myself. My preferred response to that, though, is not to build a FAI enforcing CEV, but to build a FAI that allows direct personal choice in areas where it's possible to recover from mistakes, but also provides the expert opinion as an oracle advice service.
Perfect knowledge is wonderful, sure, but was not key to my point.
Given two processes for making some decision, if process P1 is more reliable than process P2, then P1 will get me better results. That's true even if P1 is imperfect. That's true even if P2 is "ask my own brain and do what it tells me." All that is required is that P1 is more reliable than P2.
It follows that when choosing between two processes to implement my values, if I can ask one question, I should ask which process is more reliable. I should not ask which process is perfect, nor ask which process resides in my brain.
ETA: I endorse providing expert opinion, even though that deprives people of the experience of figuring it all out for themselves... agreed that far. But I also endorse providing reliable infrastructure, even though that deprives people of the experience of building all the infrastructure themselves, and I endorse implementing reliable decision matrices, even though that deprives people of the experience of making all the decisions themselves.
There's no reason you have to choose just once, a single process to answer all kinds of questions. Different processes better fit different domains. Expert opinion best fits well-understood, factual, objective, non-politicized, amoral questions. Noninterference best fits matters where people are likely to want to interfere in others' decisions and there is no pre-CEV consensus on whether such intervention is permissible.
The problem with making decisions for others isn't that it deprives them of the experience of making decisions, but that it can influence or force them into decisions that are wrong in some sense of the word.
(shrug) Letting others make decisions for themselves can also influence or force them into decisions that are wrong in some sense of the word. If that's really the problem, then letting people make their own decisions doesn't solve it. The solution to that problem is letting whatever process is best at avoiding wrong answers make the decision.
And, sure, there might be different processes for different questions. But there's no a priori reason to believe that any of those processes reside in my brain.
True. Nonintervention only works if you care about it more than about anything people might do due to it. Which is why a system of constraints that is given to the AI and is not CEV-derived can't be just nonintervention, it has to include other principles as well and be a complete ethical system.
I'm always open to suggestions of new processes. I just don't like the specific process of CEV, which happens not to reside in my brain, but that's not why I dislike it.
Ah, OK.
At the beginning of this thread you seemed to be saying that your current preferences (which are, of course, the product of a computation that resides in your brain) were the best determiner of what to optimize the environment for. If you aren't saying that, but merely saying that there's something specific about CEV that makes it an even worse choice, well, OK. I mean, I'm puzzled by that simply because there doesn't seem to be anything specific about CEV that one could object to in that way, but I don't have much to say about that; it was the idea that the output of your current algorithms are somehow more reliable than the output of some other set of algorithms implemented on a different substrate that I was challenging.
Sounds like a good place to end this thread, then.
Really? What about the "some people are Jerks" objection? That's kind of a big deal. We even got Eliezer to tentatively acknowledge the theoretical possibility that that could be objectionable at one point.
(nods) Yeah, I was sloppy. I was referring to the mechanism for extrapolating a coherent volition from a given target, rather than the specification of the target (e.g., "all of humanity") or other aspects of the CEV proposal, but I wasn't at all clear about that. Point taken, and agreed that there are some aspects of the proposal (e.g. target specification) that are specific enough to object to.
Tangentially, I consider the "some people are jerks" objection very confused. But then, I mostly conclude that if such a mechanism can exist at all, the properties of people are about as relevant to its output as the properties of states or political parties. More thoughts along those lines here.
Yes, I was talking about shortcomings of CEV, and did not mean to imply that my current preferences were better than any third option. They aren't even strictly better than CEV; I just think they are better overall if I can't mix the two.
It just seems likely, based on my understanding of what people like and approve of.
Scrict non-interference is unlikely to end up in CEV, because there are many cases where interventions are the right thing to do. I just meant it as a proof that there are less controversial principles that will block a lot of bullshit. Not as a speculation of something that will actually end up in CEV.
These values are based on false beliefs, inconsistent memes, and fear. None of those things will survive CEV. "If we knew more, thought faster, grew closer together, etc".
That would take a whole hell of a lot of certainty. I have nowhere near that level of confidence in anything I believe.
I think CEV will end up more like transhumanism than like islam. (which means I mostly accept transhumanism). I think I'm too far outside the morally-certain-but-ignorant-human reference class to make outside view judgements on this.
Not an equal amount, but many of my current values will be contradicted in CEV. I can only analogize to physics: I accept relativity, but expect it to be wrong. (I think my current beliefs are the closest approximation to CEV that I know of).
Likely candidates? That's like asking "which of your beliefs are false". All I can say is which are most uncertain. I can't say which way they will go. I am uncertain about optimal romantic organization (monogamy, polyamory, ???). I am uncertain of the moral value of closed simulations. I am uncertain about moral value of things like duplicating people, or making causally-identical models. I am quite certain that existing lives have high value. I am unsure about lives that don't yet exist.
Not quite. Let's imagine two bootstrap scenarios: some neo-enlightenment transhumanists, and some religious nuts. Even just the non-extrapolated values of the tranhumanists will produce a friendly-enough AI that can (and will want to) safey research better value-extrapolation methods. Bootstrapping it with islam will get you an angry punishing god that may or may not care about extrapolating further. Running the final, ideal CEV process with either seed should produce the same good value set, but we may not have the final ideal CEV process, and having a dangerous genie running the process may not do safe things if you start it with the wrong seed
Sorry, I made that too specific. I didn't mean to imply that only the islamists are inconsistent. Just meant them as an obvious example.
This is what I think would be good as a seed value system so that the FAI can go and block x-risk and stop death and such without having to philosophize too much first. But I'd want the CEV philosophising to be done eventually (ASAP, actually).
Right according to whose values? The problem is precisely that people disagree pre-extrapolation about when it's right to interfere, and therefore we fear their individual volitions will disagree even post extrapolation. I and some other people have a value of noninterference in certain matters that is very important to us. I would rather hardcode that value than let CEV of humanity decide on it.
Again why? CEV is very much underspecified. To me, the idea that our values and ideals will preferentially turn out to be the ones all humans would embrace "if they were smarter etc" looks like mere wishful thinking. Values are arational and vary widely. If you specify a procedure (CEV) whereby they converge to a compatible set which also happens to resemble our actual values today, then it should be possible to give different algorithms (which you can call CEV or not, it doesn't matter) which converge on other value-sets.
In the end, as the Confessor said, "you have judged: what else is there?" I have judged, and where I am certain enough about my judgement I would rather that other people's CEV not override me.
Other than that I agree with you about using a non-CEV seed etc. I just don't think we should later let CEV decide anything it likes without the seed explicitly constraining it.
CEV's. Where by an unqualified "CEV" I take nyan to be referring to CEV<humanity> ("the Coherently Extrapolated Values of Humanity"). I assume he also means it as a normative assertion of the slightly-less-extrapolated kind that means something like "all properly behaving people of my tribe would agree and if they don't we may need to beat them with sticks until they do."
And the bracketed condition is generalisable to all sorts of things - including those preferences that we haven't even considered the possibility of significant disagreement about. Partially replacing one's own preferences with preferences that are not one's own is one of the most dangerous things it is possible to do. Not something to do lightly or take for granted as implicitly 'right'.
I note that any assertion that "intervention is strictly not the wrong thing to do" that is not qualified necessarily implies a preference for the worst things that could possibly happen in an FAI-free world happening than a single disqualified intervention. That means, for example, that rather than a minimalist intervention you think the 'right' behavior for the FAI is to allow everyone on the planet to be zapped by [The Pacifier](http://en.wikipedia.org/wiki/SkyHigh(2005_film) and constantly raped by pedophiles until they are 10 whereupon they are forced to watch repeats of the first season of Big Brother until they reach 20 and are zapped again and the process is repeated until the heat death of the universe. That's pretty bad but certainly not the worst thing that could happen. It is fairly trivially not "right" to not let that happen if you can easily stop it.
Note indicating partial compatibility of positions: There can be reasons to advocate the implementation of ethical injunctions in a created GAI but that this still doesn't allow us to say that non-intervention in a given extreme circumstance is 'right'.
That's exactly what I think. And especially if you precommit to the values output by a certain process before the process is actually performed, and can't undo it later.
I'm certainly not advocating absolute unqualified non-intervention. I wrote "a value of noninterference in certain matters". Certainly the AI should interfere to e.g. offer help just before something happens which the AI thinks the person would not want to happen to them (the AI is qualified to make such decisions if it can calculate CEV). In such a situation the AI would explain matters and offer aid and advice, but ultimate deciding power might still lie with the person, depending on the circumstances.
Nonintervention doesn't just mean non-intervention by the AI, it means nonintervention by one person with another. If someone makes a request for the AI to prevent another person from doing something to them, then again in at least some (most? all?) circumstances the AI should interfere to do so; that is actually upholding the principle of uninterference.
Gah. I had to read through many paragraphs of drivel plot and then all I came up with was "a device that zaps people, making them into babies, but that is reversible". You should have just said so. (Not that the idea makes sense on any level). Anyway, my above comment applies; people would not want it done to them and so would request the AI to prevent it.
I like both these caveats. The scenario becomes something far more similar to what a CEV could plausibly be without the artificial hack. Horror stories become much harder to construct.
Off the top of my head one potential remaining weaknesses include the inability to prevent a rival, less crippled AGI from taking over without interfering pre-emptively with an individual who is not themselves interfering with anyone. Getting absolute power requires intervention (or universally compliant subjects). Not getting absolute power means something else can get it and outcomes are undefined.
That's a good point. The AI's ability to not interfere is constrained by its need to monitor everything that's going on. Not just to detect someone building a rival AI, but to detect simpler cases like someone torturing a simulated person, or even just a normal flesh and bone child who wasn't there nine months earlier. To detect people who get themselves into trouble without yet realizing it, or who are going to attack other people nonconsensually, and give these people help before something bad actually happens to them, all requires monitoring.
And while a technologically advanced AI might monitor using tools we humans couldn't even detect today, to advanced posthumans every possible tool might be painfully obvious. E.g. if you have to expose everything your megaton-of-computronium brain calculates to the AI, because that's enough to simulate all the humans alive in 2012 in enough detail that they would count as persons to the AI. But to the asteroid-sized brain this means the AI is literally aware of all its thoughts: it has zero privacy.
It does appear that universal surveillance is the cost of universally binding promises (you won't be tortured no matter where you go and what you do in AI-controlled space). To reduce costs and increase trust, the AI should be transparent to everyone itself, and should be publicly and verifiably committed to being a perfectly honest and neutral party that never reveals the secrets and private information it monitors to anyone.
I'd like to note that all of this also applies to any FAI singleton that implements some policies that we today consider morally required - like making sure no-one is torturing simulated people or raising their baby wrong. If there's no generally acceptable FAI behavior that doesn't include surveillance, then all else is equal and I still prefer my AI to a pure CEV implementation.
It would seem that the FAI should require only to be exposed to you the complete state of your brain at a point of time where it can reliably predict or prove that you are 'safe', using the kind of reasoning we often assume as a matter of course when describing UDT decision problems. Such an FAI would have information about what you are thinking - and in particular a great big class of what it knows you are not thinking - but not necessary detailed knowledge of what you are thinking specifically.
For improved privacy the inspection could be done by a spawned robot AI programmed to self destruct after analyzing you and returning nothing but a boolean safety indicator back to the FAI.
Prediction has some disadvantages compared to constant observation:
OTOH, surveillance might be much cheaper (I don't know for sure) and also allows destroying the evidence close to the site of observation once it is analyzed, preserving a measure of privacy.