Snowyowl comments on Less Wrong: Open Thread, September 2010 - Less Wrong

3 Post author: matt 01 September 2010 01:40AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (610)

Tags: open_thread

Comments (610)

Sort By: Best

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: Snowyowl 01 September 2010 11:36:18AM * 2 points [-]

I don’t care if AI is Friendly or not. [...] I am mainly interested in that whatever AI we create does not paperclip the universe

You contradict yourself here. A Friendly AI is an intelligence which attempts to improve the well-being of humanity. A paperclip maximiser is an intelligence which does not, as it cares about something different and unrelated. Any sufficiently advanced AI is either one or the other or somewhere in between.

By "sufficiently advanced", I mean an AI which is intelligent enough to consider the future of humanity and attempt to influence it.

Comment author: PhilGoetz 01 September 2010 04:37:25PM 4 points [-]

You contradict yourself here. A Friendly AI is an intelligence which attempts to improve the well-being of humanity. A paperclip maximiser is an intelligence which does not, as it cares about something different and unrelated. Any sufficiently advanced AI is either one or the other or somewhere in between.

No; these are two types of AIs out of a larger design space. You ignore, at the very least, the most important and most desirable case: An AI that shares many of humanity's values, and attempts to achieve those values rather than increase the well-being of humanity.

Comment author: billswift comment score below threshold [+] (5 children)

Comment author: Alexandros 01 September 2010 04:59:06PM 2 points [-]

You have merely redefined the goal from 'the benefit of humanity' to 'non dead-end goal', which may just be equally hairy.

Comment author: billswift 01 September 2010 05:13:29PM * 2 points [-]

Even more hairy. Any primary goal will, I think, eventually end up with a paperclipper. We need more research into how intelligent beings (ie, humans) actually function. I do not think people, with rare exceptions, actually have primary goals, only temporary, contingent goals to meet temporary ends. That is one reason I don't think much of utilitarianism - peoples "utilities" are almost always temporary, contingent, and self-limiting.

This is also one reason why I have said that I think provably Friendly AI is impossible. I will be glad to be proven wrong if it does turn out to be possible.

Comment author: Oscar_Cunningham 01 September 2010 01:15:25PM 2 points [-]

What would be a non-friendly goal that isn't dead-end? (N.B. Not a rhetorical question.)

Comment author: NihilCredo 01 September 2010 02:22:03PM 5 points [-]

Deciding that humanity was a poor choice for the dominant sapient species and should be replaced by (the improved descendants of) dolphins or octopi?

Comment author: Snowyowl 01 September 2010 01:22:55PM 2 points [-]

Ah, I see. I misunderstood your definition of "paperclip maximiser"; I assumed paperclip maximiser and Unfriendly AI were equivalent. Sorry.

Next question: if maximising paperclips or relentless self-optimisation is a dead-end goal, what is an example of a non-dead-end goal? Is there a clear border between the two, which will be obvious to the AI?

To my mind, if paperclip maximisation is a dead end, then so is everything else. The Second Law of Thermodynamics will catch up with you eventually. Nothing you create will endure forever. The only thing you can do is try to maximise your utility for as long as possible, and if that means paperclips, then so be it.

Snowyowl comments on Less Wrong: Open Thread, September 2010 - Less Wrong

Article Navigation

Comments (610)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (610)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login