Software will be superhuman good at understanding what humans mean but catastrophically worse than all previous generations at doing what humans mean.

Every bit of additional functionality requires huge amounts of HUMAN development and testing, not in order to compile and run (that's easy), but in order to WORK AS YOU WANT IT TO.

I can fully believe that a superhuman intelligence examining you will be fully capable of calculating "what you mean" "what you want" "what you fear" "what would be funniest for a buzzfeed artcle if I pretended to misunderstand your statement as meaning" "what would be best for you according to your values" "what would be best for you according to your cat's values" "what would be best for you according to Genghis Khan's values" .

No program now cares about what you mean. You've still not given any reason for the future software to care about "what you mean" over all those other calculation either.

Comment author: Juno_Watt comment score below threshold [+] (12 children)

Comment author: RobbBB 12 September 2013 06:46:39PM 6 points [-]

Software that initially appears to care what you mean will be selected by market forces. But nearly all software that superficially looks Friendly isn't Friendly. If there are seasoned AI researchers who can't wrap their heads around the five theses, then how can I be confident that the Invisible Hand will both surpass them intellectually and recurrently sacrifice short-term gains on this basis?

Comment author: TheAncientGeek 29 April 2014 03:25:35PM 0 points [-]

Software that looks friendly isn't really friendly in the sense that it really understands what we want. It isn't dangerously unfriendly because we're still here. If its commercially successful, it's friendly enough for us to want it in our lives.

Comment author: RobbBB 29 April 2014 11:10:58PM 0 points [-]

Human beings aren't friendly, in the Friendly-AI sense. If a random human acquired immense power, it would probably result in an existential catastrophe. Humans do have a better sense of human value than, say, a can-opener does; they have more power and autonomy than a can-opener, so they need fuller access to human values in order to reach similar safety levels. A superintelligent AI would require even more access to human values to reach comparable safety levels.

Comment author: TheAncientGeek 30 April 2014 12:06:21PM * 1 point [-]

There is more than one sense to friendly .AI.

If you grafted absolute power onto a human with average ethical insight, you might get absolute corruption. But what is that analogous to in .AI terms? Why assume asymmetric development by default?

If you assume top down singleton AI with a walled of ethics module, things look difficult. If you reverse this assumptions, FAI is already happening.

Comment deleted 12 September 2013 07:53:00PM * [+] (6 children)

Comment author: RobbBB 12 September 2013 08:57:06PM 1 point [-]

Nearly all software that superficially looks like it's going to go skynet on you and kill you, isn't going to do that, either.

Sure. Because nearly all software that superficially looks to a human like it's a seed AI is not a seed AI. The argument for 'programmable indirect normativity is an important research focus' nowhere assumes that it's particularly easy to build a seed AI.

"If there are seasoned AI researchers who can't wrap their heads around the five theses", then you are going to feel more pleased with yourself, being a believer

Hm? No. Dissonance is painful. People feel happier agreeing than disagreeing.

which releases dopamine and reinforces what ever fallacies of reasoning you make.

Releasing dopamine also reinforces whatever correct reasoning one carries out. Good reasoning is just as much a brain process as bad reasoning.

Comment deleted 12 September 2013 09:22:41PM * [-]

Comment author: RobbBB 12 September 2013 09:38:15PM * 1 point [-]

I didn't say everyone who rejects any of the theses does so purely because s/he didn't understand it. That doesn't make it cease to be a problem that most AGI researchers don't understand all of the theses, or the case supporting them. You may be familiar with the theses only from the Sequences, but they've all been defended in journal articles, book chapters, and conference papers. See e.g. Chalmers 2010 and Chalmers 2012 for the explosion thesis, or Bostrom 2012 for the orthogonality thesis.

Comment deleted 12 September 2013 10:12:33PM * [-]

Comment author: RobbBB 13 September 2013 08:05:45AM * 0 points [-]

You're still picking those particular views due to the endorsement by Yudkowsky.

Your psychological speculation fails you. I actually read the articles I cited, and I found their arguments convincing.

With regards to Chalmers and Bostrom, they are philosophers with zero understanding of the actual issues involved in AI

This makes it sound like you've never read anything by those two authors on the subject. Possibly you're trying to generalize from your cached idea of a 'philosopher'. Expertise in philosophy does not in itself make one less qualified to weigh in on meta-ethics, normative ethics, epistemology, philosophy of mind, moral psychology, philosophy of cognitive and information sciences, or for that matter AI theory. Read the papers I cited. They get the empirical facts right, they focus on the issues philosophy is more or less built for, and they present their case clearly and concisely. If philosophers have no place in this debate, then they have no place in any debate.

that guy has quite distinct understanding of the whole issue from the others

What's the distinction you have in mind?

which is probably why you wouldn't list him.

No? I've cited Omohundro before, and I'll cite him again.

Juno_Watt comments on The genie knows, but doesn't care - Less Wrong

Article Navigation

Comments (515)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (515)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login