Why would an AI try to figure out its goals?

cousin_it

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

I can think of several good reasons why values might not be incorporated into a system as "ought" beliefs. If my AI isn't very good at reasoning, I might, for instance, find it simpler to construct a black-box "does this action have consequence X" property-checker and incorporate that into the system somewhere. The rest of the system has no access to the internals of the black box -- it just supplies a proposed course of action and gets back a YES or a NO.

You ask whether there's "any AI research ongoing that would support the notion that AIs wouldn't have some sort of systematic categorization of beliefs and values?"

Most of what's currently published at major AI research conferences describes systems that don't have any such systematic characterization. Suppose we built a super-duper Watson that passed the Turing test and had some limited capacity to improve itself by, e.g., going out and fetching new information from the Internet. That soft of system strikes me as the likeliest one to meet the bar of "AGI" in the next few years. It isn't particularly far from current research.

Before you quibble about whether that's the kind of system we're talking about -- I haven't seen a good definition of "self-improving" program, and I suspect it is not at all straightforward to define. Among other reasons, I don't know a good definition that separates 'code' and 'data'. So if you don't like the example above, you should make sure that there's a clear difference between choosing what inputs to read (which modifies internal state) and choosing what code to load (which also modifies internal state).

As to the human example: my sense is that humans don't get locked to any one set of goals; that goals continue to evolve, without much careful pruning, over a human lifetime. Expecting an AI to tinker with its goals for a while, and then stop, is asking it to do something that neither natural intelligences or existing software seems to do or even be capable of doing.

Suppose we built a super-duper Watson that passed the Turing test and had some limited capacity to improve itself by, e.g., going out and fetching new information from the Internet. That sort of system strikes me as the likeliest one to meet the bar of "AGI" in the next few years. It isn't particularly far from current research.

This seems like a plausible way of blowing up the universe, but not in the next few years. This kind of thing requires a lot of development, I'd give it 30-60 years at least.

-2Logos0115y

... I think we're having a major breakdown of communication because to my understanding Watson does exactly what you just claimed no AI at research conferences is doing. [...] I'm sure. But there's a few generally sound assertions we can make: 1. To be self-improving the machine must be able to examine its own code / be "metacognitive." 2. To be self-improving the machine must be able to produce a target state. From these two the notion of value fixation in such an AI would become trivial. Even if that version of the AI would have man-made value-fixation, what about the AI it itself codes? If the AI were actually smarter than us, that wouldn't exactly be the safest route to take. Even Asimov's Three Laws yielded a Zeroth Law. [...] 1. Don't anthropomorphize. :) 2. If you'll recall from my description, I have no such expectation. Instead, I spoke of recursive refinement causing apparent fixation in the form of "gravity" or "stickiness" towards a specific set of values. Why is this unlike how humans normally are? Well, we don't have much access to our own actual values.

21

Why would an AI try to figure out its goals?

21

21

21

Why would an AI try to figure out its goals?

21

21