You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

David_Gerard comments on AI Risk and Opportunity: A Strategic Analysis - Less Wrong Discussion

8 Post author: lukeprog 04 March 2012 06:06AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (161)

Tags: singularity ai

Comments (161)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: David_Gerard 04 March 2012 02:36:45PM * 3 points [-]

Has anyone constructed even a vaguely plausible outline, let alone a definition, of what would constitute a "human-friendly intelligence", defined in terms other than effects you don't want it to have? As you note, humans aren't human-friendly intelligences, or we wouldn't have internal existential risk.

The CEV proposal seems to attempt to move the hard bit to technological magic (a superintelligence scanning human brains and working out a solution to human desires that is possible, is coherent and won't destroy us all) - this is saying "then a miracle occurs" in more words.

Comment author: John_Maxwell_IV 04 March 2012 11:51:25PM 2 points [-]

As you note, humans aren't human-friendly intelligences, or we wouldn't have internal existential risk.

It's possible that particular humans might approximate human friendly intelligences.

Comment author: David_Gerard 05 March 2012 08:02:55AM -1 points [-]

Assuming it's not impossible, how would you know? What constitutes a human-friendly intelligence, in other than negative terms?

Comment author: timtyler 05 March 2012 08:01:45PM -2 points [-]

Has anyone constructed even a vaguely plausible outline, let alone a definition, of what would constitute a "human-friendly intelligence", defined in terms other than effects you don't want it to have?

Er, that's how it is defined - at least by Yudkowsky. You want to argue definitions? Without even offering one of your own? How will that help?

Comment author: David_Gerard comment score below threshold [+] (3 children)

Comment author: timtyler 05 March 2012 08:43:24PM * 1 point [-]

Yudkowsky says:

The term "Friendly AI" refers to the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals.

That isn't a "purely negative" definition in the first place.

Even if it was - would you object to the definition of "hole" on similar grounds?

What exactly is wrong with defining some things in terms of what they are not?

It I say a "safe car" is one that doesn't kill or hurt people, that seems just fine to me.

Comment author: David_Gerard 05 March 2012 10:44:32PM * 0 points [-]

The word "artificial" there makes it look like it means more than it does. And humans are just as made of atoms. Let's try it without that:

The term "friendly intelligence" refers to the production of human-benefiting, non-human-harming actions in intelligences that have advanced to the point of making real-world plans in pursuit of goals.

It's only described in terms of its effects, and then only vaguely. We have no idea what it would actually be. The CEV plan doesn't include what it would actually be, it just includes a technological magic step where it's worked out.

This may be better than nothing, but it's not enough to say it's talking about anything that's actually understood in even the vaguest terms.

For an analogy, what would a gorilla-friendly human-level intelligence be like? How would you reasonably make sure it wasn't harmful to the future of gorillas? (Humans out the box do pretty badly at this.) What steps would the human take to ascertain the CEV of gorillas, assuming tremendous technological resources?

Comment author: timtyler 06 March 2012 12:13:15AM 0 points [-]

We can't answer the "how can you do this?" questions today. If we could we would be done.

It's true that CEV is an 8-year old, moon-onna-stick wishlist - apparently created without much thought about to how to implement it. C'est la vie.

David_Gerard comments on AI Risk and Opportunity: A Strategic Analysis - Less Wrong Discussion

Article Navigation

Comments (161)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Open Threads

Recent Rationality Diaries

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (161)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login