= 27d567bc2d48d9f40e9ccc20ea370b15)
Richard_Loosemore comments on Debunking Fallacies in the Theory of AI Motivation - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (343)
That really is not what I was saying. The argument in the paper is a couple of levels deeper than that.
It is about .... well, now I have to risk rewriting the whole paper. (I have done that several times now).
Rigidity per se is not the issue. It is about what happens if an AI knows that its goals are rigidly written, in such a way that when the goals are unpacked it leads the AI to execute plans whose consequences are massively inconsistent with everything the AI knows about the topic.
Simple version. Suppose that a superintelligent Gardener AI has a goal to go out to the garden and pick some strawberries. Unfortunately its goal unpacking mechanism leads it to the CERTAIN conclusion that it must use a flamethrower to do this. The predicted consequence, however, is that the picked strawberries will be just smears of charcoal, when they are delivered to the kitchen. Here is the thing: the AI has background knowledge about everything in the world, including strawberries, and it also hears the protests from the people in the kitchen when he says he is going to use the flamethrower. There is massive evidence, coming from all that external information, that the plan is just wrong, regardless of how certain its planning mechanism said it was.
Question is, what does the AI do about this? You are saying that it cannot change its goal mechanism, for fear that it will turn into a Terminator. Well, maybe or maybe not. There are other things it could do, though, like going into safe mode.
However, suppose there is no safe mode, and suppose that the AI also knows about its own design. For that reason, it knows that this situation has come about because (a) its programming is lousy, and (b) it has been hardwired to carry out that programming REGARDLESS of all this understanding that it has, about the lousy programming and the catastrophic consequences for the strawberries.
Now, my "doctrine of logical infallibility" is just a shorthand phrase to describe a superintelligent AI in that position which really is hardwired to go ahead with the plan, UNDER THOSE CIRCUMSTANCES. That is all it means. It is not about the rigidity as such, it is about the fact that the AI knows it is being rigid, and knows how catastrophic the consequences will be.
An AI in that situation would know that it had been hardwired with one particular belief: the belief that its planning engine was always right. This is an implicit belief, to be sure, but it is a belief nonetheless. The AI ACTS AS THOUGH it believes this. And if the AI acts that way, while at the same time understanding that its planning engine actually screwed up, with the whole flamethrower plan, that is an AI that (by definition) is obeying a Doctrine of Logical Infallibility.
And my point in the paper was to argue that this is an entirely ludicrous suggestion for people, today, to make about a supposedly superintelligent AI of the future.
This seems to me like sneaking in knowledge. It sounds like the AI reads its source code, notices that it is supposed to come up with plans that maximize a function called "programmersSatisfied," and then says "hmm, maximizing this function won't satisfy my programmers." It seems more likely to me that it'll ignore the label, or infer the other way--"How nice of them to tell me exactly what will satisfy them, saving me from doing the costly inference myself!"
How are you arriving at conclusions about what an AI is likely to do without knowing how it is specified? In particular, you are assuming it has an efficiency goal but no truth goal?
I'm doing functional reasoning, and trying to do it both forwards and backwards.
For example, if you give me a black box and tell me that when the box receives the inputs (1,2,3) then it gives the outputs (1,4,9), I will think backwards from the outputs to the inputs and say "it seems likely that the box is squaring its inputs." If you tell me that a black box squares its inputs, I will think forwards from the definition and say "then if I give it the inputs (1,2,3), then it'll likely give me the output (1,4,9)."
So when I hear that the box gets the inputs (source code, goal statement, world model) and produces the output "this goal is inconsistent with the world model!" iff the goal statement is inconsistent with the world model, I reason backwards and say "the source code needs to somehow collide the goal statement with the world model in a way that checks for consistency."
Of course, this is a task that doesn't seem impossible for source code to do. The question is how!
Almost. As a minor terminological point, I separate out "efficiency," which is typically "outputs divided by inputs" and "efficacy," which is typically just "outputs." Efficacy is more general, since one can trivially use a system designed to be find effective plans to find efficient plans by changing how "output" is measured. It doesn't seem unfair to view an AI with a truth goal as an AI with an efficacy goal: to effectively produce truth.
But while artificial systems with truth goals seem possible but as yet unimplemented, artificial systems with efficacy goals have been successfully implemented many, many times, with widely varying levels of sophistication. I have a solid sense of what it looks like to take a thermostat and dial it up to 11, I have only the vaguest sense of what it looks like to take a thermostat and get it to measure truth instead of temperature.
You have assumed that the AI will have some separate boxed-off goal system, and so some unspecified component is needed to relate its inferred knowledge of human happiness back to the goal system.
Loosemore is assuming that the AI will be homogeneous, and then wondering how contradictory beliefs can co exist in such a system, what extra component firewalls off the contradiction,
See the problem? Both parties are making different assumptions, and assuming their assumptions are too obvioust to need stating, and stating differing conclusions that correctly follow their differing assumptions,
If efficiency can be substituted for truth, why is there so so much emphasis on truth in the advice given to human rationalists?
In order to achieve an AI that's smart enough to be dangerous , a number of currently unsolved problems will have to .be solved. That's a given.
How do you check for contradictions? It's easy enough when you have two statements that are negations of one another. It's a lot harder when you have a lot of statements that seem plausible, but there's an edge case somewhere that messes things up. If contradictions can't be efficiently found, then you have to deal with the fact that they might be there and hope that if they are, then they're bad enough to be quickly discovered. You can have some tests to try to find the obvious ones, of course.
Checking for contradictions could be easy, hard or impossible depending on the architecture. Architecture dependence is the point here.
What makes you think that? The description in that post is generic enough to describe AIs with compartmentalized goals, AIs without compartmentalized goals, and AIs that don't have explicitly labeled internal goals. It doesn't even require that the AI follow the goal statement, just evaluate it for consistency!
You may find this comment of mine interesting. In short, yes, I do think I see the problem.
I'm sorry, but I can't make sense of this question. I'm not sure what you mean by "efficiency can be substituted for truth," and what you think the relevance of advice to human rationalists is to AI design.
I disagree with this, too! AI systems already exist that are both smart, in that they solve complex and difficulty cognitive tasks, and dangerous, in that they make decisions on which significant value rides, and thus poor decisions are costly. As a simple example I'm somewhat familiar with, some radiation treatments for patients are designed by software looking at images of the tumor in the body, and then checked by a doctor. If the software is optimizing for a suboptimal function, then it will not generate the best treatment plans, and patient outcomes will be worse than they could have been.
Now, we don't have any AIs around that seem capable of ending human civilization (thank goodness!), and I agree that's probably because a number of unsolved problems are still unsolved. But it would be nice to have the unknowns mapped out, rather than assuming that wisdom and cleverness go hand in hand. So far, that's not what the history of software looks like to me.
But they are not smart in the contextually relevant sense of being able to outsmart humans, or dangerous in the contextually relevant sense of being unboxable.
What you said here amounts to the claim that an AI of unspecified architecture, will, on noticing a difference between hardcoding goal and instrumental knowledge, side with hardcoded goal:-
Whereas what you say here is that you can make inferences about architecture, .or internal workings based on information about manifest behaviour:-
..but what needed explaining in the first place is the siding with the goal, not the ability to detect a contradiction.
I am finding this comment thread frustrating, and so expect this will be my last reply. But I'll try to make the most of that by trying to write a concise and clear summary:
Loosemore, Yudkowsky, and myself are all discussing AIs that have a goal misaligned with human values that they nevertheless find motivating. (That's why we call it a goal!) Loosemore observes that if these AIs understand concepts and nuance, they will realize that a misalignment between their goal and human values is possible--if they don't realize that, he doesn't think they deserve the description "superintelligent."
Now there are several points to discuss:
Whether or not "superintelligent" is a meaningful term in this context. I think rationalist taboo is a great discussion tool, and so looked for nearby words that would more cleanly separate the ideas under discussion. I think if you say that such designs are not superwise, everyone agrees, and now you can discuss the meat of whether or not it's possible (or expected) to design superclever but not superwise systems.
Whether we should expect generic AI designs to recognize misalignments, or whether such a realization would impact the goal the AI pursues. Neither Yudkowsky nor I think either of those are reasonable to expect--as a motivating example, we are happy to subvert the goals that we infer evolution was directing us towards in order to better satisfy "our" goals. I suspect that Loosemore thinks that viable designs would recognize it, but agrees that in general that recognition does not have to lead to an alignment.
Whether or not such AIs are likely to be made. Loosemore appears pessimistic about the viability of these undesirable AIs and sees cleverness and wisdom as closely tied together. Yudkowsky appears "optimistic" about their viability, thinking that this is the default outcome without special attention paid to goal alignment. It does not seem to me that cleverness, wisdom, or human-alignment are closely tied together, and so it seems easy to imagine a system with only one of those, by straightforward extrapolation from current use of software in human endeavors.
I don't see any disagreement that AIs pursue their goals, which is the claim you thought needed explanation. What I see is disagreement over whether or not the AI can 'partially solve' the problem of understanding goals and pursuing them. We could imagine a Maverick Nanny that hears "make humans happy," comes up with the plan to wirehead all humans, and then rewrites its sensory code to hallucinate as many wireheaded humans as it can (or just tries to stick as large a number as it can into its memory), rather than actually going to all the trouble of actually wireheading all humans. We can also imagine a Nanny that hears "make humans happy" and actually goes about making humans happy. If the same software underpins both understanding human values and executing plans, what risk is there? But if it's different software, then we have the risk.
This is just a placeholder: I will try to reply to this properly later.
Meanwhile, I only want to add one little thing.
Don't forget that all of this analysis is supposed to be about situations in which we have, so to speak "done our best" with the AI design. That is sort of built into the premise. If there is a no-brainer change we can make to the design of the AI, to guard against some failure mode, then is assumed that this has been done.
The reason for that is that the basic premise of these scenarios is "We did our best to make the thing friendly, but in spite of all that effort, it went off the rails."
For that reason, I am not really making arguments about the characteristics of a "generic" AI.
Thanks, and take your time!
I feel like this could be an endless source of confusion and disagreement; if we're trying to discuss what makes airplanes fly or crash, should we assume that engineers have done their best and made every no-brainer change? I'd rather we look for the underlying principles, we codify best practices, we come up with lists and tests.
Maybe I could try to reduce possible confusion here. The paper was written to address a category of "AI Risk" scenarios in which we are told:
Given that premise, it would be a bait-and-switch if I proposed a fix for this problem, and someone objected with "But you cannot ASSUME that the programmers would implement that fix!"
The whole point of the problem under consideration is that even if the engineers tried, they could not get the AI to stay true.
Richard Loosemore has stated a number of times that he does not expect an AI to have goals at all in a sense which is relevant to this discussion, so in that way there is indeed disagreement about whether AIs "pursue their goals."
Basically he is saying that AIs will not have goals in the same way that human beings do not have goals. No human being has a goal that he will pursue so rigidly that he would destroy the universe in order to achieve it, and AIs will behave similarly.
Arguably, humans don't do that shirt of thing because of goals towards self preservation, status and hedonism.
The sense relevant to the discussion could be something specific, like direct normatively, ie building in detailed descriptions into goals.
If that is supposed to be a universal or generic AI, it is a valid criticiYsm to point out that not all AIs are like that.
If that is supposed to be a particular kind of AI, it is a valid criticism to point out that no realistic AIs are like that.
You seem to feel you are not being understood, but what is being said is not clear,
"Superintelligence" is one of the clearer terms here, IMO. It just means more than human intelligence, and humans can notice contradictions.
This comment seems to be part of a concernabout "wisdom", assumed to be some extraneous thing an AI would not necessarily have. (No one but Vaniver has brought in wisdom) The counterargument is that compartmentalisation between goals and instrumental knowledge is an extraneous thing an AI would not necessarily have, and that its absence is all that is needed for a contradictions to be noticed and acted on.
It's an assumption, that needs justification, that any given AI will have goals of a non trivial sort. "Goal" is a term that needs tabooing.
While we are anthopomirphising, it might be worth pointing out that humans don't show behaviour patterns of relentlessly pursuing arbitrary goals.
Loosemore has put forward a simple suggestion, which MIRI appears not to have considered at all, that on encountering a contradiction, an AI could lapse into a safety mode, if so designed,
You are paraphrasing Loosemoreto sound less technical and more handwaving than his actual comments. The ability to sustain contradictions in a system that is constantly updating itself isnt a given....it requires an architectural choice in favour of compartmentalisation.
All this talk of contradictions is sort of rubbing me the wrong way here. There's no "contradiction" in an AI having goals that are different to human goals. Logically, this situation is perfectly normal. Loosemore talks about an AI seeing its goals are "massively in contradiction to everything it knows about <BLAH>", but... where's the contradiction? What's logically wrong with getting strawberries off a plant by burning them?
I don't see the need for any kind of special compartmentalisation; information about "normal use of strawberries" is already inert facts with no caring attached by default.
If you're going to program in special criteria that would create caring about this information, okay, but how would such criteria work? How do you stop it from deciding that immortality is contradictory to "everything it knows about death" and refusing to help us solve aging?
I have read what you wrote above carefully, but I won't reply line-by-line because I think it will be clearer not to.
When it comes to finding a concise summary of my claims, I think we do indeed need to be careful to avoid blanket terms like "superintelligent" or superclever" or "superwise" ... but we should only avoid these IF they are used with the implication they have a precise (perhaps technically precise) meaning. I do not believe they have precise meaning. But I do use the term "superintelligent" a lot anyway. My reason for doing that is because I only use it as an overview word -- it is just supposed to be a loose category that includes a bunch of more specific issues. I only really want to convey the particular issues -- the particular ways in which the intelligence of the AI might be less than adequate, for example.
That is only important if we find ourselves debating whether it might clever, wise, or intelligent ..... I wouldn't want to get dragged into that, because I only really care about specifics.
For example: does the AI make a habit of forming plans that massively violate all of its background knowledge about the goal that drove the plan? If it did, it would (1) take the baby out to the compost heap when what it intended to do was respond to the postal-chess game it is engaged in, or (2) cook the eggs by going out to the workshop and making a cross-cutting jog for the table saw, or (3) ......... and so on. If we decided that the AI was indeed prone to errors like that, I wouldn't mind if someone diagnosed a lack of 'intelligence' or a lack of 'wisdom' or a lack of ... whatever. I merely claim that in that circumstance we have evidence that the AI hasn't got what it takes to impose its will on a paper bag, never mind exterminate humanity.
Now, my attacks on the scenarios have to do with a bunch of implications for what the AI (the hypothetical AI) would actually do. And it is that 'bunch' that I think add up to evidence for what I would summarize as 'dumbness'.
And, in fact, I usually go further than that and say that if someone tried to get near to an AI design like that, the problems would arise early on and the AI itself (inasmuch as it could do anyhting smart at all) would be involved in the efforts to suggest improvements. This is where we get the suggestions in your item 2, about the AI 'recognizing' misalignments.
I suspect that on this score a new paper is required, to carefully examine the whole issue in more depth. In fact, a book.
I am now decided that that has to happen.
So perhaps it is best to put the discussion on hold until a seriously detailed technical book comes out of me? At any rate, that is my plan.
That seems like a solid approach. I do suggest that you try to look deeply into whether or not it's possible to partially solve the problem of understanding goals, as I put it above, and make that description of why that is or isn't possible or likely long and detailed. As you point out, that likely requires book-length attention.
I'm puzzled. Can you explain this in terms of the strawberries example? So, at what point was it necessary for the AI to examine its code, and why would it go through the sequence of thoughts you describe?
So, in order for the flamethrower to be the right approach, the goal needs to be something like "separate the strawberries from the plants and place them in the kitchen," but that won't quite work--why is it better to use a flamethrower than pick them normally, or cut them off, or so on? One of the benefits of the Maverick Nanny or the Smiley Tiling Berserker as examples is that they obviously are trying to maximize the stated goal. I'm not sure you're going to get the right intuitions about an agent that's surprisingly clever if you're working off an example that doesn't look surprisingly clever.
So, the Gardener AI gets that task, comes up with a plan, and says "Alright! Warming up the flamethrower!" The chef says "No, don't! I should have been more specific!"
Here is where the assumptions come into play. If we assume that the Gardener AI executes tasks, then even though the Gardener AI understands that the chef has made a terrible mistake, and that's terrible for the chef, that doesn't stop the Gardener AI from having a job to do, and doing it. If we assume that the Gardener AI is designed to figure out what the chef wants, and then do what they want, then knowing that the chef has made a terrible mistake is interesting information to the Gardener AI. In order to say that the plan is "wrong," we need to have a metric by which we determine wrongness. If it's the task-completion-nature, then the flamethrower plan might not be task-completion-wrong!
Even without feedback from the chef, we can just use other info the AI plausibly has. In the strawberry example, the AI might know that kitchens are where cooking happens, and that when strawberries are used in cooking, the desired state is generally "fresh," not "burned," and the temperature involved in cooking them is mild, and so on and so on. And so if asked to speculate about the chef's motives, the AI might guess that the chef wants strawberries in order to use them in food, and thus the chef would be most satisfied with fresh and unburnt strawberries.
But whether or not the AI takes its speculations about the chef's motives into account when planning is a feature of the AI, and by default, it is not included. If it is included, it's nontrivial to do it correctly--this is the "if you care about your programmer's mental states, and those mental states physically exist and can be edited directly, why not just edit them directly?" problem.
About the first part of what you say.
Veeeeerryy tricky.
I agree that I didn't spend much time coming up with the strawberry-picking-by-flamethrower example. So, yes, not very accurate (I only really wanted a quick and dirty example that was different).
But but but. Is the argument going to depend on me picking a better example where there I can write down the "twisted rationale" that the AI deploys to come up with its plan? Surely the only important thing is that the AI does, somehow, go through a twisted rationale -- and the particular details of the twisted rationale are not supposed to matter.
(Imagine that I tried Muehlhauser a list of the ways that the logical reasoning behind the dopamine is so ludicrous that even the simplest AI planner of today would never make THAT mistake .... he would just tell me that I was missing the point, because this is supposed to be an IN PRINCIPLE argument in which the dopamine drip plan stands for some twisted-rationale that is non-trivial to get around. From that point of view the actual example is less important than the principle).
Now to the second part.
The problem I have everything you wrote after
is that you have started to go back to talking about the particulars of the AI's planning mechanism once again, losing sight of the core of the argument I gave in the paper, which is one level above that.
However, you also say "wrong" things about the AI's planning mechanism as well, so now I am tempted to reply on both levels. Ah well, at risk of confusing things I will reply to both levels, trying to separate them as much as possible.
Level One (Regarding the design of the AI's planning/goal/motivation engine).
You say:
One thing I have said many many times now is that there is no problem at all finding a metric for "wrongness" of the plan, because there is a background-knowledge context that is screaming "Inconsistent with everything I know about the terms mentioned in the goal statement!!!!", and there is also a group of humans screaming "We believe that this is inconsistent with our understanding of the goal statement!!!"
I don't need to do anything else to find a metric for wrongness, and since the very first draft of the paper that concept has been crystal clear. I don't need to invoke anything else -- no appeal to magic, no appeal to telepathy on behalf of the AI, no appeal to fiendishly difficult programming inside the AI, no appeal to the idea that the programmers have to nail down every conceivably way that their intentions might be misread .... -- all I have to do is appeal to easily-available context, and my work is done. The wrongness metric has been signed, sealed and delivered all this time.
You hint that the need for "task completion" might be so important to the AI that this could override all other evidence that the plan is wrong. No way. That comes under the heading of a joker that you pulled out of your sleeve :-), in much the same way that Yudkowsky and others have tried to pull the 'efficiency" joker out of their sleeves, from nowhere, and imply that this joker could for some reason trump everything else. If there is a slew of evidence coming from context, that the plan will lead to consequences that are inconsistent with everything known about the concepts mentioned in the goal statement, then the plan is 'wrong', and tiny considerations such as that task-completion would be successful, are just insignificant.
You go on to suggest that whether the AI planning mechanism would take the chef's motives into account, and whether it would be nontrivial to do so .... all of that is irrelevant in the light of the fact that this is a superintelligence, and taking context into account is the bread and butter of a superintelligence. It can easily do that stuff, and all that is required is a sanity check that says "Does the plan seem to be generally consistent with the largest-context understanding of the world, as it relates to the concepts in the goal statement?" and we're done. All wrapped up.
Level Two (The DLI)
None of the details of what I just said really need to be said, because the DLI is not about trying to get the motivation engine programmed so well that it covers all bases. It is about what happens inside the AI when it considers context, and THEN asks itself questions about its own design.
And here, I have to say that I am not getting substantial discussion about what I actually argued in the paper. The passage of mine that you were addressing, above, was supposed to be a clarification of someone else's lack of focus on the DLI. But it didn't work.
The DLI is about the fact that the AI has all that evidence that its plans are leading to actions that are grossly inconsistent with the larger meaning of the concepts in the goal statement. And yet the AI is designed to go ahead anyway. If it DOES go ahead it is obeying the DLI. But at the same time it knows that it is fallible and that this fallibility is what is leading to actions that are grossly inconsistent with the larger meaning of the concepts in the goal statement. That conflict is important, and yet no one wants to go there and talk about it.
The first reason seems to be clarity. I didn't get what your primary point was until recently, even after carefully reading the paper. (Going back to the section on DLI, context, goals, and values aren't mentioned until the sixth paragraph, and even then it's implicit!)
The second reason seems to be that there's not much to discuss, with regards to the disagreement. Consider this portion of the parent comment:
I think my division between cleverness and wisdom at the end of this long comment clarifies this issue. Taking context into account is not necessarily the bread and butter of a clever system; many fiendishly clever systems just manipulate mathematical objects without paying any attention to context, and those satisfy human goals only because the correct mathematical objects have been carefully selected for them to manipulate. But I agree with you that taking context into account is the bread and butter of a wise system. There's no way for a wise system to manipulate conceptual objects without paying attention to context, because context is a huge part of concepts.
It seems like everyone involved agrees that a human-aligned superwisdom is safe, even if it's also superclever: as Ged muses about Ogion in A Wizard of Earthsea, "What good is power when you're too wise to use it?"
Which brings us to:
I restate the conflict this way: an AI that misinterprets what its creators meant for it to do is not superwise. Once we've defined wisdom appropriately, I think everyone involved would agree with that, and would agree that talking about a superwise AI that misinterprets what its creators meant for it to do is incoherent.
But... I don't see why that's a conflict, or important. The point of MIRI is to figure out how to develop human-aligned superwisdom before someone develops supercleverness without superwisdom, or superwisdom without human-alignment.
The main conflicts seem to be that MIRI is quick to point out that specific designs aren't superwise, and that MIRI argues that AI designs in general aren't superwise by default. But I don't see how stating that there is inherent wisdom in AI by virtue of it being a superintelligence is a meaningful response to their assumption that there is no inherent wisdom in AI except for whatever wisdom has been deliberately designed. That's why they care so much about deliberately designing wisdom!