Why would an AI try to figure out its goals?

cousin_it

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

Of the seventy-some definitions of intelligence that had been gathered last count, most have something to do with achieving goals. That is a very different thing from being goal-directed (which has several additional requirements, the most obvious being an explicit representation of one's goals).

OK, so I am not 100% clear on the distinction you are trying to draw - but I just mean optimising, or maximising.

Would you murder your next-door neighbor if you thought you could get away with it?

Hmm - so: publicly soliciting personally identifiable expressions of murderous intent is probably not the best way of going about this. If it helps, I do think that Skinnerian conditioning - based on punishment and reprimands - is the proximate explanation for most avoidance of "bad" actions.

It remains the case that most creatures are not particularly goal-directed. We know that bees stockpile honey to survive the winter, but the bees do not know this.

So: the bees are optimised to make more bees. Stockpiling honey is part of that. Knowing why is not needed for optimisation.

Even the most intelligent animals have planning horizons of minutes compared to lifespans of years to decades.

OK - but even plants are optimising. There are multiple optimisation processes. One happens inside minds - that seems to be what you are talking about. Mindless things optimise too though - plants act so as to maximise the number of their offspring - and that's still a form of optimisation.

If you want the rationale for describing such actions as being "goal directed", we can consider the goal to be world domination by the plants, and then the actions of the plant are directed towards that goal. You can still have "direction" without a conscious "director".

Hmm - so: publicly soliciting personally identifiable expressions of murderous intent is probably not the best way of going about this

It was a rhetorical question. I'm confident the answer is no - the law only works when most people are basically honest. We think we have a goal, and so we do by the ordinary English meaning of the word, but then there are things we are not prepared to do to achieve it, so it turns out what we have is not a goal by the ultimate criterion of decision theory on which Omohundro draws, and if we try to rescue the overuse of d... (read more)

21

Why would an AI try to figure out its goals?

21

21

21

Why would an AI try to figure out its goals?

21

21