Why would an AI try to figure out its goals?

cousin_it

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

This sentence seems off. It isn't clear what is meant by mechanical in this context other than to shove through a host of implied connotations.

Hrm. If I had used the word "procedural" rather than "mechanical", would that have, do you think, prevented this impression?

Asserting that something non-trivial can be put in terms of math when one can't do so on one's own and doesn't provide a reference seems less than conducive to good discussion.

If I am not a physicist, does that disqualify me from making claims about what a physicist would be relatively easily able to do? For example; "I'm not sufficient to the task of calculating my current relativistic mass -- but anyone who works with general relativity would have no trouble doing this."

So what am I missing with this element? Because I genuinely cannot see a difference between "a mathematician / AI worker could express in mathematical or computational terms the nature of recursive selection pressure" and "a general relativity physicist could calculate my relativistic mass relative to the Earth" in terms of the exceptionalism of either claim.

Is it perhaps that my wording appears to be implying that I meant more than "goals can be arranged in a graph of interdependent nodes that recursively update one another for weighting"?

Part of the reason why the sentence bothers me is that I'm a mathematician and it wasn't obvious to me that there is a useful way of making the statement mathematically precise.

Is it perhaps that my wording appears to be implying that I meant more than "goals can be arranged in a graph of interdependent nodes that recursively update one another for weighting"?

So this is a little better and that may be part of it. Unfortunately, it isn't completely obvious that this is true either. This is a property that we want goal systems to have in some form. It isn't obvious that all goal systems in some broad sense will necessarily do so.

21

Why would an AI try to figure out its goals?

21

21

21

Why would an AI try to figure out its goals?

21

21