Why would an AI try to figure out its goals?

cousin_it

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?

"So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, The Basic AI Drives

This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, What I Think, If Not Why

I have stopped understanding why these quotes are correct. Help!

When you talk about AI, do you mean general intelligence, as in being competent in arbitrary domains (given enough computing power), or narrow AI, which can succeed on some classes of tasks but fail on others? I would certainly agree that narrow AI does not need to be goal-directed, and the future will surely contain many such AI. And maybe there are ways to achieve general intelligence other than through a goal-directed architecture, but since that's already fairly simple, and all of our theories and existing examples point towards it, it just seems very unlikely that the first AGI that we build won't goal-directed.

Given enough computing power, humans can create a haphazardly smart jumble of wires by simulated evolution

So far, evolution has created either narrow intelligence (non-human animals) or general intelligence that is goal-directed. Why would simulated evolution give different results?

uploading small chunks of human brains and prodding them

It seems to me that you would again end up with either a narrow intelligence or a goal-directed general intelligence.

I see no reason why all such creatures would necessarily have an urge to stabilize their values.

Again, if by AI you include narrow AI, then I'd agree with you. So what question are you asking?

BTW, an interesting related question is whether general intelligence is even possible at all, or can we only build AIs that are collections of tricks and heuristics, and we ourselves are just narrow intelligence with competence in enough areas to seem like general intelligence. Maybe that's the question you actually have in mind?

What is the difference between what you mean by "goal-directed AGI" and "not goal-directed AGI", given that the latter is stipulated as "competent in arbitrary domains (given enough computing power)"? What does "competent" refer to in the latter, if not to essentially goal-directedness, that is successful attainment of whatever "competence" requires by any means necessary (consequentialism, means don't matter in themselves)? I think these are identical ideas, and rightly so.

1cousin_it15y

I don't know how to unpack "general intelligence" or "competence in arbitrary domains" and I don't think people have any reason to believe they possess something so awesome. When people talk about AGI, I just assume they mean AI that's at least as general as a human. A lobotomized human is one example of a "jumble of wires" that has human-level IQ but scores pretty low on goal-directedness. The first general-enough AI we build will likely be goal-directed if it's simple and built from first principles. But if it's complex and cobbled together from "shallow insights", its goal-directedness and goal-stabilization tendencies are anyone's guess.

21

Why would an AI try to figure out its goals?

21

21

21

Why would an AI try to figure out its goals?

21

21