A paperclipper no more has a wall stopping it from updating into morality than my laptop has a wall stopping it from talking to me. My laptop doesn't talk to me because I didn't program it to. You do not update into pushing pebbles into prime-numbered heaps because you're not programmed to do so.

Does a stone roll uphill on a whim?

Perhaps you should study Reductionism first.

Comment author: PrawnOfFate comment score below threshold [+] (11 children)

Comment author: Nornagest 24 April 2013 02:19:21AM 4 points [-]

Not programmed to, or programmed not to? If you can code up a solution to value drift, lets see it. Otherwise, note that Life programmes can update to implement glider generators without being "programmed to".

...with extremely low probability. It's far more likely that the Life field will stabilize around some relatively boring state, empty or with a few simple stable patterns. Similarly, a system subject to value drift seems likely to converge on boring attractors in value space (like wireheading, which indeed has turned out to be a problem with even weak self-modifying AI) rather than stable complex value systems. Paperclippism is not a boring attractor in this context, and a working fully reflective Clippy would need a solution to value drift, but humanlike values are not obviously so, either.

Comment author: PrawnOfFate 25 April 2013 12:34:28PM * 1 point [-]

I'm increasingly baffled as to why AI is always brought in to discussions of metaethics. Societies of rational agents need ethics to regulate their conduct. Out AIs aren't sophisticated enough to live in their own socieities. A wireheading AI isn't even going to be able to survive "in the wild". If you could build an artificial society of AI, then the questions of whether they spontaneously evolved ethics would be a very interesting and relevant datum. But AIs as we know them aren't good models for the kinds of entities to which morality is relevant. And Clippy is particularly exceptional example of an AI. So why do people keep saying "Ah, but Clippy..."...?

Comment author: Nornagest 25 April 2013 06:45:38PM * 2 points [-]

And Clippy is particularly exceptional example of an AI. So why do people keep saying "Ah, but Clippy..."...?

Well, in this case it's because the post I was responding to mentioned Clippy a couple of times, so I thought it'd be worthwhile to mention how the little bugger fits into the overall picture of value stability. It's indeed somewhat tangential to the main point I was trying to make; paperclippers don't have anything to do with value drift (they're an example of a different failure mode in artificial ethics) and they're unlikely to evolve from a changing value system.

Comment author: MugaSofer comment score below threshold [+] (6 children)

Comment author: PrawnOfFate comment score below threshold [+] (5 children)

Comment author: PrawnOfFate 24 April 2013 02:23:49AM -2 points [-]

At last, an interesting reply!

PrawnOfFate comments on Welcome to Less Wrong! (5th thread, March 2013) - Less Wrong

Article Navigation

Comments (1750)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (1750)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login