NancyLebovitz comments on Only humans can have human values - Less Wrong

34 Post author: PhilGoetz 26 April 2010 06:57PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (159)

Comments (159)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 27 April 2010 04:06:13PM 6 points [-]

Banal translation:

The implied argument is that there aren't any values at all that most people will agree on, because one imagined and not-evolutionarily-viable Clippy doesn't think anything other than paperclips have value.

No, that is not the argument implied when making references to paperclipping. That is a silly argument that is about a whole different problem to paperclipping. It is ironic that your straw man claim is, in fact, the straw man.

But it would seem our disagreement if far more fundamental than what a particular metaphor means:

one imagined and not-evolutionarily-viable Clippy

  1. Being "Evolutionarily-viable" is a relatively poor form of optimisation. It is completely the wrong evaluation of competitiveness to make and also carries the insidious assumption that competing is something that an agent should do as more than a short term instrumental objective.
  2. Clippy is competitively viable. If you think that a Paperclip Maximizer isn't a viable competitive force then you do not understand what a Paperclip Maximizer is. It maximizes paperclips. It doesn't @#%$ around making paperclips while everyone else is making Battle Cruisers and nanobots. It kills everyone, burns the cosmic commons to whatever extent necessary to eliminate any potential threat and then it goes about turning whatever is left into paperclips.
  3. The whole problem with Paperclip Maximisers is that they ARE competitively viable. That is the mistake in the design. A mandate to produce a desirable resource (stationary) will produce approximately the same behavior as a mandate to optimise survival, dominance and power right up until the point where it doesn't need to any more.
Comment author: NancyLebovitz 27 April 2010 04:25:18PM 0 points [-]

t kills everyone, burns the cosmic commons to whatever extent necessary to eliminate any potential threat and then it goes about turning whatever is left into paperclips.

That's where Clippy might fail at viability-- unless it's the only maximizer around, that "kill everyone" strategy might catch the notice of entities capable of stopping it-- entities that wouldn't move against a friendlier AI.

A while ago, there was some discussion of AIs which cooperated by sharing permission to view source code. Did that discussion come to any conclusions?

Assuming it's possible to verify that the real source code is being seen, I don't think a paper clipper isn't going to get very far unless the other AIs also happen to be paper clippers.

Comment author: JGWeissman 27 April 2010 07:06:29PM 3 points [-]

That's where Clippy might fail at viability-- unless it's the only maximizer around, that "kill everyone" strategy might catch the notice of entities capable of stopping it-- entities that wouldn't move against a friendlier AI.

An earth originating paperclipper that gets squashed by other super intelligences from somewhere else still is very bad for humans.

Though I don't see why a paperclipper couldn't compromise and cooperate with competing super intelligences as well as other super intelligences with different goals. If other AIs are a problem for Clippy, they are also a problem for AIs that are Friendly towards humans, but not neccesarily friendly towards alien super intelligences.

Comment author: wedrifid 28 April 2010 05:34:45AM 1 point [-]

That's where Clippy might fail at viability-- unless it's the only maximizer around, that "kill everyone" strategy might catch the notice of entities capable of stopping it-- entities that wouldn't move against a friendlier AI.

Intended to be a illustration of how Clippy can do completely obvious things that don't happen to be stupid, not a coded obligation. Clippy will of course do whatever is necessary to gain more paper-clips. In the (unlikely) event that Clippy finds himself in a situation in which cooperation is a better maximisation strategy than simply outfooming then he will obviously cooperate.

Comment author: NancyLebovitz 28 April 2010 10:00:22AM 0 points [-]

It isn't absolute not-viability, but the odds are worse for an AI which won't cooperate unless it sees a good reason to do so than for an AI which cooperates unless it sees a good reason to not cooperate.

Comment author: wedrifid 28 April 2010 06:59:31PM 4 points [-]

but the odds are worse for an AI which won't cooperate unless it sees a good reason to do so than for an AI which cooperates unless it sees a good reason to not cooperate.

Rationalists win. Rational paperclip maximisers win then make paperclips.

Comment author: cwillu 27 April 2010 06:30:22PM 0 points [-]

Fair point, but the assumption that it indeed is possible to verify source code is far from proven. There's too many unknowns in cryptography to make strong claims as to what strategies are possible, let alone which would be successful.

Comment author: NancyLebovitz 27 April 2010 06:38:20PM 1 point [-]

And we've got to assume AIs would be awfully good at steganography.

Powered by Reddit Powered by Reddit