Why would an AI try to figure out its goals?

cousin_it

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

There are many humans who haven't thought seriously about their goals in life.

They're not intelligent on a large scale. That is, knowing how they act on the short term would be a more effective way to find out how they end up than knowing what their goals are. They still may have short-term intelligence, in that knowing how they act on the very short term would be less useful to predicting what happens than what they want on the short term. For example, you'd be much better off predicting the final state of a chess game against someone who's much better than you at chess by how they want it to end than by what their moves are.

It's not impossible to accomplish your goals without knowing what they are, but it helps. If you're particularly intelligent in regards to some goal, you'd probably do it. Nobody knows their goals exactly, but they generally know it enough to be helpful.

It would be a very counterintuitive definition of "intelligent" that said that a machine that thought and communicated as well as a very-well-read and high-IQ 13-year-old wasn't intelligence.

Very few possible conversations sound remotely like a very-well-read high-IQ 13-year-old. Anything that can do it is either a very good optimization process, or just very complicated (such as a recording of such a conversation).

Edit:

I suppose I should probably justify why this definition of intelligence is useful here. When we try to build an AI, what we want is something that will accomplish our goals. It will make sure that, of all possible universes, the one that happens is particularly high on our utility ranking. In short, it must be intelligent with respect to our goals by the definition of intelligence I just gave. If it can hold a conversation like a 13-year-old, but can't do anything useful, we don't want it.

21

Why would an AI try to figure out its goals?

21

21

21

Why would an AI try to figure out its goals?

21

21