Why would an AI try to figure out its goals?

cousin_it

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

Most of what's currently published at major AI research conferences describes systems that don't have any such systematic characterization. Suppose we built a super-duper Watson

... I think we're having a major breakdown of communication because to my understanding Watson does exactly what you just claimed no AI at research conferences is doing.

Before you quibble about whether that's the kind of system we're talking about -- I haven't seen a good definition of "self-improving" program, and I suspect it is not at all straightforward to define.

I'm sure. But there's a few generally sound assertions we can make:

To be self-improving the machine must be able to examine its own code / be "metacognitive."
To be self-improving the machine must be able to produce a target state.

From these two the notion of value fixation in such an AI would become trivial. Even if that version of the AI would have man-made value-fixation, what about the AI it itself codes? If the AI were actually smarter than us, that wouldn't exactly be the safest route to take. Even Asimov's Three Laws yielded a Zeroth Law.

Expecting an AI to tinker with its goals for a while, and then stop,

Don't anthropomorphize. :)
If you'll recall from my description, I have no such expectation. Instead, I spoke of recursive refinement causing apparent fixation in the form of "gravity" or "stickiness" towards a specific set of values.

Why is this unlike how humans normally are? Well, we don't have much access to our own actual values.

21

Why would an AI try to figure out its goals?

21

21

21

Why would an AI try to figure out its goals?

21

21