= 27d567bc2d48d9f40e9ccc20ea370b15)
Kenny comments on Magical Categories - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (89)
It is just me, or are things getting a bit unfriendly around here?
Anyway...
Wiring up the AI to maximise happy faces etc. is not a very good idea, the goal is clearly too shallow to reflect the underlying intent. I'd have to read more of Hibbard's stuff to properly understand his position, however.
That said, I do agree with a more basic underlying theme that he seems to be putting forward. In my opinion, a key, perhaps even THE key to intelligence is the ability to form reliable deep abstractions. In Solomonoff induction and AIXI you see this being driving by the Kolmogorov compressor, in the brain the neocortical hierarchy seems to be key. Furthermore, if you adopt the perspective I've taken on intelligence (i.e. the universal intelligence measure) you see that the reverse implication is true: intelligence actually requires the ability to form deep abstractions. In which case, a *super* intelligent machine must have the ability to form *very* deep and reliable abstractions about the world. Such a machine could still try to turn the world into happy faces, if this was its goal. However, it wouldn't do this by accident because its ability to form abstractions was so badly flawed that it doesn't differentiate between smiling faces and happy people. It's not that stupid. Note that this goes for forming powerful abstractions in general, not just human things like happiness and faces.
"It's not that stupid."
What if it doesn't care about happiness or smiles or any other abstractions that we value? A super-intelligence isn't an unlimited intelligence, i.e. it would still have to choose what to think about.
I think the point is that if you accept this definition of intelligence, i.e. that it requires the ability to form deep and reliable abstractions about the world, then it doesn't make sense to talk about any intelligence (let alone a super one) being unable to differentiate between smiley-faces and happy people. It isn't a matter, at least in this instance, of whether it cares to make that differentiation or not. If it is intelligent, it will make the distinction. It may have values that would be unrecognizable or abhorrent to humans, and I suppose that (as Shane_Legg noted) it can't be ruled out that such values might lead it to tile the universe with smiley-faces, but such an outcome would have to be the result of something other than a mistake. In other words, if it really is "that stupid," it fails in a number of other ways long before it has a chance to make this particular error.
I wrote a post about this! See The genie knows, but doesn't care.
It may not make sense to talk about a superintelligence that's too dumb to understand human values, but it does make sense to talk about an AI smart enough to program superior general intelligences that's too dumb to understand human values. If the first such AIs ('seed AIs') are built before we've solved this family of problems, then the intelligence explosion thesis suggests that it will probably be too late. You could ask an AI to solve the problem of FAI for us, but it would need to be an AI smart enough to complete that task reliably yet too dumb (or too well-boxed) to be dangerous.
Superior to what? If they are only as smart as the average person, then all things being equal, they will be as good as the average peson as figuring out morality. If they are smarter, they will be better, You seem to be tacitly assuming that the Seed AIs are designing walled-off unupdateable utility functions. But if one assumes a more natural architecture, where moral sense is allowed to evolve with eveythign else, you would expect and incremental succession of AIs to gradually get better at moral reasoning. And if it fooms, it's moral reasoning will fomm along with eveything else, because you haven't created an artificial problem by firewalling it off.
Superior to itself.
That's not generally true of human-level intelligences. We wouldn't expect a random alien species that happens to be as smart as humans to be very successful at figuring out human morality. It maybe true if the human-level AGI is an unmodified emulation of a human brain. But humans aren't very good at figuring out morality; they can make serious mistakes, though admittedly not the same mistakes Eliezer gives as examples above. (He deliberately picked ones that sound 'stupid' to a human mind, to make the point that human concepts have a huge amount of implicit complexity built in.)
Not necessarily. The average chimpanzee is better than the average human at predicting chimpanzee behavior, simulating chimpanzee values, etc. (See Sympathetic Minds.)
Utility functions that change over time are more dangerous than stable ones, because it's harder to predict how a descendant of a seed AI with a heavily modified utility function will behave than it is to predict how a descendant with the same utility function will behave.
If we don't solve the problem of Friendly AI ourselves, we won't know what trajectory of self-modification to set the AI on in order for it to increasingly approximate Friendliness. We can't tell it to increasingly approximate something that we ourselves cannot formalize and cannot point to clear empirical evidence of.
We already understand arithmetic, so we know how to reward a system for gradually doing better and better at arithmetic problems. We don't understand human morality or desire, so we can't design a Morality Test or Wish Test that we know for sure will reward all and only the good or desirable actions. We can make the AI increasingly approximate something, sure, but how do we know in advance that that something is something we'd like?
Assuming morality is lots of highly localised, different things...which I don't , particularly. if it is not, then you can figure it out anywhere, If it is,then the problem the aliens have is not that morality is imponderable, but that they are don't have access to the right data. They don't know how things on earth. However, an AI built on Earth woud. So the situation is not analgous. The only disadvantage an AI would have is not having biological drives itself, but it is not clear that an entity needs to have drives in order to understand them. We could expect a SIAI to get incrementally betyter at maths than us until it surpasses us; we wouldn't worry that i would hit on the wrong maths, because maths is not a set of arbitrary, disconnected facts.
An averagely intelligent AI with an average grasp of morlaity would not be more of a threat than an average human. A smart AI, would, all other things being equal, be better at figuring out moralitry. But all other things are not equal, because you want to create problems by wallign off the UF.
I'm sure they do. That seems to be why progress in AGI , specifically use of natural language,has been achingly slow. But why should moral concepts be som much more difficult than others? An AI smart enought to talk its way out of a box would be able to understand the implicit complexity: an AI too dumb to understand implicit complexity would be boxable. Where is the problem?
Things are not inherently dagerous just because they are unpredictable. If you have some independent reason fo thinking something might turn dangerous, then it becomes desirable to predict it.
But Superintelligent artifical general intelligences are generally assumed to be good at eveything: they are not assumed to develop mysterious blind spots about falconry or mining engineering, Why assume they will develop a blind spot about morality? Oh yes...because you have assumed from the outset that the UF must be walled off from self mprovement...in order to be safe. You are only facing that particualr failure mode because of something you decided on to be safe.
The average person manages to solve the problem of being moral themselves, in a good-enough way. You keep assuming, without explanation that an AI can't do the same.
Why isn't havign a formalisation of morality a prolem with humans? We know how humans incremently improve as moral reasoners: it's called the Kohlberg hierarchy.
We don't have perfect morality tests. We do have morality tests. Fail them and you get pilloried in the media or sent to jail.
Again, you are assumign that morality is something highly local and arbitrary. If it works like arithmetic, that is if it is an expansion of some basic principles, then we can tell that is heading in the right direction by identifying ithat its reasoning is inline with those principles.
The problem of FAI is the problem of figuring out all of humanity's deepest concerns and preferences, not just the problem of figuring out the 'moral' ones (whichever those are). E.g., we want a superintelligence to not make life boring for everyone forever, even if 'don't bore people' isn't a moral imperative.
Regardless, I don't see how the moral subset of human concerns could be simplified without sacrificing most human intuitions about what's right and wrong. Human intuitions as they stand aren't even consistent, so I don't understand how you can think the problem of making them consistent and actionable is going to be a simple one.
Someday, perhaps. With enough time and effort invested. Still, again, we would expect a lot more human-intelligence-level aliens (even if those aliens knew a lot about human behavior) to be good at building better AIs than to be good at formalizing human value. For the same reason, we should expect a lot more possible AIs we could build to be good at building better AIs than to be good at formalizing human value.
I don't know what you mean by 'imponderable'. Morality isn't ineffable; it's just way too complicated for us to figure out. We know how things are on Earth; we've been gathering data and theorizing about morality for centuries. And our progress in formalizing morality has been minimal.
An AI that's just a copy of a human running on transistors is much more powerful than a human, because it can think and act much faster.
It would also be better at figuring out how many atoms are in my fingernail, but that doesn't mean it will ever get an exact count. The question is how rough an approximation of human value can we allow before all value is lost; this is the 'fragility of values' problem. It's not enough for an AGI to do better than us at FAI; it has to be smart enough to solve the problem to a high level of confidence and precision.
First, because they're anthropocentric; 'iron' can be defined simply because it's a common pattern in Nature, not a rare high-level product of a highly contingent and complex evolutionary history. Second, because they're very inclusive; 'what humans care about' or 'what humans think is Right' is inclusive of many different human emotions, intuitions, cultural conventions, and historical accidents.
But the main point is just that human value is difficult, not that it's the most difficult thing we could do. If other tasks are also difficult, that doesn't necessarily make FAI easier.
You're forgetting the 'seed is not the superintelligence' lesson from The genie knows, but doesn't care. If you haven't read that article, go do so. The seed AI is dumb enough to be boxable, but also too dumb to plausibly solve the entire FAI problem itself. The superintelligent AI is smart enough to solve FAI, but also too smart to be safely boxed; and it doesn't help us that an unFriendly superintelligent AI has solved FAI, if by that point it's too powerful for us to control. You can't safely pass the buck to a superintelligence to tell us how to build a superintelligence safe enough to pass bucks to.
Yes. The five theses give us reason to expect superintelligent AI to be dangerous by default. Adding more unpredictability to a system that already seems dangerous will generally make it more dangerous.
'The genie knows, but doesn't care' means that the genie (i.e., superintelligence) knows how to do human morality (or could easily figure it out, if it felt like trying), but hasn't been built to care about human morality. Knowing how to behave the way humans want you to is not sufficient for actually behaving that way; Eliezer makes that point well in No Universally Compelling Arguments.
The worry isn't that the superintelligence will be dumb about morality; it's that it will be indifferent to morality, and that by the time it exists it will be too late to safely change that indifference. The seed AI (which is not a superintelligence, but is smart enough to set off a chain of self-modifications that lead to a superintelligence) is dumb about morality (approximately as dumb as humans are, if not dumber), and is also probably not a particularly amazing falconer or miner. It only needs to be a competent programmer, to qualify as a seed AI.
Good enough for going to the grocery store without knifing anyone. Probably not good enough for safely ruling the world. With greater power comes a greater need for moral insight, and a greater risk should that insight be absent.
It is a problem, and it leads to a huge amount of human suffering. It doesn't mean we get everything wrong, but we do make moral errors on a routine basis; the consequences are mostly non-catastrophic because we're slow, weak, and have adopted some 'good-enough' heuristics for bounded circumstances.
Just about every contemporary moral psychologist I've read or talked to seems to think that Kohlberg's overall model is false. (Though some may think it's a useful toy model, and it certainly was hugely influential in its day.) Haidt's The Emotional Dog and Its Rational Tail gets cited a lot in this context.
That's certainly not good enough. Build a superintelligence that optimizes for 'following the letter of the law' and you don't get a superintelligence that cares about humans' deepest values. The law itself has enough inexactness and arbitrariness that it causes massive needless human suffering on a routine basis, though it's another one of those 'good-enough' measures we keep in place to stave off even worse descents into darkness.
Human values are an evolutionary hack resulting from adaptations to billions of different selective pressures over billions of years, innumerable side-effects of those adaptations, genetic drift, etc. Arithmetic can be formalized in a few sentences. Why think that humanity's deepest preferences are anything like that simple? Our priors should be very low for 'human value is simple' just given the etiology of human value, and our failure to converge on any simple predictive or normative theory thus far seems to only confirm this.
It's quite possible that I'm below average, but I'm not terribly impressed by my own ability to extrapolate how other average people's morality works -- and that's with the advantage of being built on hardware that's designed toward empathy and shared values. I'm pretty confident I'm smarter than my cat, but it's not evident that I'm correct when I guess at the cat's moral system. I can be right, at times, but I can be wrong, too.
Worse, that seems a fairly common matter. There are several major political discussions involving moral matters, where it's conceivable that at least 30% of the population has made an incorrect extrapolation, and probable that in excess of 60% has. And this only gets worse if you consider a time variant : someone who was as smart as the average individual in 1950 would have little problem doing some very unpleasant things to Alan Turing. Society (luckily!) developed since then, but it has mechanisms for development and disposal of concepts that AI do not necessarily have or we may not want them to have.
((This is in addition to general concerns about the universality of intelligence : it's not clear that the sort of intelligence used for scientific research necessarily overlaps with the sort of intelligence used for philosophy, even if it's common in humans.))
Well, the obvious problem with not walling off and making unupdateaable the utility function is that the simplest way to maximize the value of a malleable utility function is to update it to something very easy. If you tell an AI that you want it to make you happy, and let it update that utility function, it takes a good deal less bit-twiddling to define "happy" as a steadily increasing counter. If you're /lucky/, that means your AI breaks down. If not, it's (weakly) unfriendly.
You can have a higher-level utility function of "do what I mean", but not only is that harder to define, it has to be walled off, or you have "what I mean" redirected to a steadily increasing counter. And so on and so forth through higher levels of abstraction.
Thanks for the reply, Robb. I've read your post and a good deal of the discussion surrounding it.
I think I understand the general concern, that an AI that either doesn't understand or care about our values could pose a grave threat to humanity. This is true on its face, in the broad sense that any significant technological advance carries with it unforeseen (and therefore potentially negative) consequences. If, however, the intelligence explosion thesis is correct, then we may be too late anyway. I'll elaborate on that in a moment.
First, though, I'm not sure I see how an AI "too dumb to understand human values" could program a superior general intelligence (i.e. an AI that is smart enough to understand human values). Even so, assuming it is possible, and assuming it could happen on a timescale and in such a way as to preclude or make irrelevant any human intervention, why would that change the nature of the superior intelligence from being, say, friendly to human interests, to being hostile to them? Why, for that matter, would any superintelligence (that understands human values, and that is "able to form deep and reliable abstractions about the world") be predisposed to any particular position vis-a-vis humans? And even if it were predisposed toward friendliness, how could we possibly guarantee it would always remain so? How, that is, having once made a friend, can we foolproof ourselves against betrayal? My intuition is that we can’t. No step can be taken without some measure of risk, however small, and if the step has potentially infinitely negative consequences, then even the very slightest of risks begins to look like a bad bet. I don’t know a way around that math.
The genie, as you say, doesn't care. But also, often enough, the human doesn't care. He is constrained, of course, by his fellow humans, and by his environment, but he sometimes still manages (sometimes alone, sometimes in groups) to sow massive horror among his fellows, sometimes even in the name of human values. Insanity, for instance, in humans, is always possible, and one definition of insanity might even be: behavior that contradicts, ignores or otherwise violates the values of normal human society. “Normal” here is variable, of course, for the simple reason that “human society” is also variable. That doesn’t stop us, however, from distinguishing, as we generally do, between the insane and the merely stupid, even if upon close inspection the lines begin to blur. Likewise, we occasionally witness - and very frequently we imagine (comic books!) - cases where a human is both super-intelligent and super-insane. The fear many people have with regard to strong AI (and it is perhaps well-grounded, or well-enough), is that it might be both super-intelligent and, at least as far as human values are concerned, super-insane. As an added bonus, and certainly if the intelligence explosion thesis is correct, it might also be unconstrained or, ultimately, unconstrainable. On this much I think we agree, and I assume the goal of FAI is precisely to find the appropriate constraints.
Back now, though, to the question of “too late.” The family of problems you propose to solve before the first so-called seed AIs are built include, if I understand you correctly, a formal definition of human values. I doubt very much that such a solution is possible - and “never” surely won’t help us any more than “too late” - but what would the discovery of (or failure to discover) such a solution have to do with a mistake such as tiling the universe with smiley-faces (which seems to me much more a semantic error than an error in value judgment)? If we define our terms - and I don’t know any definition of intelligence that would allow the universe-tiling behavior to be called intelligent - then smiley faces may still be a risk, but they are not a risk of intelligent behavior. They are one way the project could conceivably fail, but they are not an intelligent failure.
On the other hand, the formal-definition-of-human-values problem is related to the smiley faces problem in another way: any hard-coded solution could lead to a universe of bad definitions and false equivalencies (smiles taken for happiness). Not because the AI would make a mistake, but because human values are neither fixed nor general nor permanent: to fix them (in code), and then propagate them on the enormous scale the intelligence explosion thesis suggests, might well lead to some kind of funneling effect, perhaps very quickly, perhaps over a long period of time, that produces, effectively, an infinity of smiley faces. In other words, to reduce an irreducible problem doesn’t actually solve it. For example, I value certain forms of individuality and certain forms of conformity, and at different times in my life I have valued other and even contradictory forms of individuality and other and even contradictory forms of conformity. I might even, today, call certain of my old individualistic values conformist values, and vice-versa, and not strictly because I know more today than I knew then. I am, today, quite differently situated in the world than I was, say, twenty years ago; I may even be said to be somewhat of a different person (and yet still the same); and around me the world itself has also changed. Now, these changes, these changing and contradictory values may or may not be the most important ones, but how could they be formalized, even conceptually? There is nothing necessary about them. They might have gone the other way around. They might not have changed at all. A person can value change and stability at the same time, and not only because he has a fuzzy sense of what those concepts mean. A person can also have a very clear idea of what certain concepts mean, and those concepts may still fail to describe reality. They do fail, actually, necessarily, which doesn’t make them useless - not at all - but knowledge of this failure should at least make us wary of the claims we produce on their behalf.
What am I saying? Basically, that the pre-seed hard-coding path to FAI looks pretty hopeless. If strong AI is inevitable, then yes, we must do everything in our power to make it friendly; but what exactly is in our power, if strong AI (which by definition means super-strong, and super-super-strong, etc.) is inevitable? If the risks associated with strong AI are as grave as you take them to be, does it really seem better to you (in terms of existential risk to the human race) for us to solve FAI - which is to say, to think we’ve solved it, since there would be no way of testing our solution “inside the box” - than to not solve strong AI at all? And if you believe that there is just no way to halt the progress toward strong AI (and super, and super-super), is that compatible with a belief that “this kind of progress” can be corralled into the relatively vague concept of “friendliness toward humans”?
Better stop there for the moment. I realize I’ve gone well outside the scope of your comment, but looking back through some of the discussion raised by your original post, I found I had more to say/think about than I expected. None of the questions here are meant to be strictly rhetorical, a lot of this is just musing, so please respond (or not) to whatever interests you.