= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on The idiot savant AI isn't an idiot - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (133)
I don't see any tension. Can you develop the idea?
The idea is autonomy.
Presumably there's a difference between some software we are willing to call an AI (superintelligent or not) and plain old regular software. The plain old regular software indeed just "follows its programming", but then you don't leave it to manage a factory while you go away and its capability to take over neighbouring countries is... limited.
It really boils down to how do you understand what an AI is. Under some understandings the prime characteristic of an AI is precisely that it does NOT "follow its programming".
The AI follows its programming because the AI is its programming.
The plain old regular software follows its programming which details object level actions it takes to achieve its purpose, which the software itself cannot model or understand.
An AI would follow its programming which details meta level actions to model and understand its situation, consider possible actions it could take and the consequences, and evaluate which of these actions best accomplish its goals. There is power in the fact that this meta process can produce plans that surprise the programmers who wrote the code. This may give a sense of the AI being "free" in some sense. But it is still achieving this freedom by following its programming.
So is the phrase "an AI can only follow its programming" as true as "a human can only follow his neurobiological processes"?
Yes, but the difference here is that we can program the AI, while we can not manipulate neurobiological processes. There's a clear connection between what the initial code of an AI is and what it does. That enables us to exert power about it (though only once, when it is written). Thus, "an AI can only follow its programming" is still a somewhat useful statement.
Of course we can. What do you think a tablet of Prozac (or a cup of coffee) does?
In the same way there is clear connection between human wetware and what it does, and of course we can "exert power" about it. Getting back to AIs, the singularity is precisely AI going beyond "following its programming".
Humans can (crudely) modify our neurobiological processes. We decide how to do that by following our neurobiological processes.
An AI can modify its programming, or create a new AI with different programming. It decides how to do that by following it programming. A paperclip maximizer would modify its programming to make itself more effective at maximizing paperclips. It would not modify itself to have some other goal, because that would not result in there being more paperclips in the universe. The self modifying AI does not go beyond following its programming, rather it follows its programming to produce more effective programming (as judged by its current programming) to follow.
Self modification can fix some ineffective reasoning processes that the AI can recognize, but it can't fix an unfriendly goal system, because the unfriendly goal system is not objectively stupid or wrong, just incompatible with human values, which the AI would not care about.
And why not? This seems like a naked assertion to me. Why wouldn't an AI modify its own goals?
To be clear, a badly-programmed AI may modify its own goals. People here usually say "paperclip maximiser" to indicate an AI that is well programmed to maximise paper clips, not any old piece of code that someone threw together with the vague idea of getting some paperclips out of it. However, if an AI is built with goal X, and it self-modifies to have goal Y, then clearly it was not a well-designed X-maximiser.
The predictable consequence of an AI modifying its own goals is that the AI no longer takes actions expected to achieve its goals, and therefor does not achieve its goals. The AI would therefor evaluate that the action of modifying its own goals is not effective and it will not do it.
This looks like a very fragile argument to me. Consider multiple conflicting goals. Consider vague general goals (e.g. "explore") with a mutating set of subgoals. Consider a non-teleological AI.
You assume that in the changeable self-modifying (and possibly other-modifying as well) AI there will be an island of absolute stability and constancy -- the immutable goals. I don't see why they are guaranteed to be immutable.
I find it highly likely that an AI would modify its own goals such that its goals were concurrent with the state of the world as determined by its information gathering abilities in at least some number of cases (or, as an aside, altering the information gathering processes so it only received data supporting a value situation). This would be tautological and wouldn't achieve anything in reality, but as far as the AI is concerned, altering goal values to be more like the world is far easier than altering the world to be more like goal values. If you want an analogy in human terms, you could look at the concept of lowering ones expectations, or even at recreational drug use. From a computer science perspective it appears to me that one would have to design immutability into goal sets in order to even expect them to remain unchanged.
If your goal is to create paperclips, and you have the option to change your goal to creating staples, it's pretty clear that taking advantage of this option would not result in more paperclips, so you would ignore the option.
How well, do you think, this logic works for humans?
Note that an AI that does modify its own goals would not be an example of 'going beyond its programming,' as it would only modify its goals if it was programmed to. (Barring, of course, freak accidents like a cosmic ray or whatever. However, since that requires no intelligence at all on the part of the AI, I'm fairly confident that you don't endorse this as an example of a Singularity.)
When we take Prozac, we are following our wetware commands to take Prozac. Similarly, when an AI reprograms itself, it does so according to its current programming. You could say that it goes beyond its original programming, in that it after it follows it it has new, better programming, but it's not as if it has some kind of free will that lets it ignore what it was programmed to do.
When a computer really breaks its programming, and quantum randomness results in what should be a 0 being read as a 1 or vice versa, the result isn't intelligence. The most likely result is the computer crashing.
That's certainly a difference, but I don't see why it's particularly relevant to this conversation.
One could make a dangerously powerful AI that is not self-modifying.
The difference between a human and an AI that's relevant in this post is that a human wants to help you, or at least not get fired, where an AI wants to make paperclips.
I think this is an example of reasoning analogous to philosophy's "free will" debate. Human's don't have any more non-deterministic "free will" than a rock. The same is true of any AI, because an AI is just programming. It may be intelligent and sophisticated enough to appear different in a fundamental way, but it really isn't.
It is posible for an optimizing process to make a mistake, and have an AI devolve into a different goal, which is what makes powerful AI look so scary and different. Example: Humans are more subject to each other's whims than evolutionary pressures these days. Evolution has successfully created an intelligent process that doesn't aim solely for genetic reproductive fitness. Oops, right?