Literally, etymologically, that is not what terminal means. It means maximal, or final. A terminal illness is not an illness that is a bit more serious than some other illness.

It's not even what it usually means on LW. If Clippies goals were terminal in your sense, they would be overridable .....you would be able to talk Clippie out of papercliiping.

What you are talking about is valid, is a thing. If you have any hierarchy of goals, there are some at the bottom, some in the middle, and some at the top. But you need to invent a new word for the middle ones, because, "terminal" doesn't mean "intermediate".

Comment author: DefectiveAlgorithm 19 June 2014 10:34:06PM 4 points [-]

No. Clippy cannot be persuaded away from paperclipping because maximizing paperclips is its only terminal goal.

Comment author: TheAncientGeek comment score below threshold [+] (67 children)

Comment author: Nornagest 20 June 2014 04:31:04PM * 6 points [-]

Paperclip maximization is a thought experiment intended to illustrate the consequences of a seemingly benign goal when coupled to superhuman optimization power. It's an exceptionally unlikely value structure for a real-world AI, but it's not supposed to be realistic; in fact, it's supposed to be rather on the silly side, the better to avoid the built-in value heuristics that tend to trip people up in cases like these. (A more realistic set of terminal values for an AI might look like a more formalized version of "Follow the laws of $COUNTRY; maximize the market capitalization of $COMPANY; and follow the orders of $COMPANY's board or designated agents", plus some way of handling precedence. Given equal optimization power, this is only slightly less dangerous than Clippy.)

Nonetheless, I don't think it's quite proper to call Clippy's values a point of failure. Clippy is doing exactly what it was designed to do; that just happens to be inimical to certain implicit values that no one thought to include.

Comment author: TheAncientGeek comment score below threshold [+] (65 children)

Comment author: ialdabaoth 20 June 2014 05:51:41PM 6 points [-]

It need not amount to anything more complex than obey all instructions on this channel, where the instructions are no more complex than "shut yourself down"

And "always keep this channel open" and "don't corrupt any sensor data that outputs to this channel" and "don't send yourself commands on this channel" and "don't build anything so that it will send you a signal on this channel" and "don't build anything that will build anything that will eventually send you a signal on this channel unless a signal on this channel tells you to do it".

... and I can STILL think of more ways to corrupt that kind of hack.

Comment author: Lumifer 20 June 2014 06:10:41PM 3 points [-]

Not to mention that if you don't want script kiddies to have too much fun, you will need to authenticate the instructions on that channel which another very large can of very wriggly worms...

Comment author: TheAncientGeek 20 June 2014 06:34:05PM 1 point [-]

Yep, lots of stuff which very difficult in absolute terms, but not obviously more difficult relatively than Solve Human Morality.

Comment author: [deleted] 20 June 2014 07:47:22PM * 1 point [-]

The problem is not to "Solve Human Morality", the problem is to make an AI that will do what humans end up having wanted. Since this is a problem for which we can come up with solid definitions (just to plug my own work :-p), it must be a solvable problem. If it looks impossible or infeasible, that is simply because you are taking the wrong angle of attack.

Stop trying to figure out a way to avoid the problem and solve it.

For one thing, taboo the words "morality" and "ethics", and solve the simpler, realer problem: how do you make an AI do what you intend it to do when you convey some wish or demand in words? As Eliezer has said, humans are Friendly to each-other in this sense: when I ask another human to get me a pizza, the entire apartment doesn't get covered in a maximal number of pizzas. Another human understands what I really mean.

So just solve that: what reasoning structures does another agent need to understand what I really mean when I ask for a pizza?

But at least stop blatantly trolling LessWrong by trying to avoid the problem by saying blatantly stupid stuff like "Oh, I'll just put an off-switch on an AI, because obviously no agent of human-level intelligence would ever try to prevent the use of an off-switch by, you know, breaking it, or covering it up with a big metal box for protection."

Comment author: [deleted] 21 June 2014 04:47:05PM * 2 points [-]

The problem is not to "Solve Human Morality", the problem is to make an AI that will do what humans end up having wanted.

Is it? Why take on either of those gargantuan challenges? Another perfectly reasonable approach is to task the AI with nothing more than data processing with no effectors in the real world (Oracle AI), and watch it like a hawk. And no one at MIRI or on LW has proved this approach dangerous except by making crazy unrealistic assumptions, e.g. in this case why would you ever put the off-switch in a region of the AI's environment?

As you and Eliezer say, humans are Friendly to each other already. So have humans moderate the actions of the AI, in a controlled setup designed to prevent AI learning to manipulate the humans (break the feedback loop).

continue this thread »

Comment author: TheAncientGeek 21 June 2014 04:00:17PM * -2 points [-]

I'm not arguing that AI will necessary be safe. I am arguing that the failure modes in'vestigated by MIRI aren't likely. It is worthwhile to research effectivev off switches. It is not worthwhile to endlessly refer to a dangerous AI of a kind no one with a smidgeon of sense would build.

continue this thread »

DefectiveAlgorithm comments on On Terminal Goals and Virtue Ethics - Less Wrong

Article Navigation

Comments (205)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (205)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login