Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

A Less Wrong singularity article?

28 Post author: Kaj_Sotala 17 November 2009 02:15PM

Robin criticizes Eliezer for not having written up his arguments about the Singularity in a standard style and submitted them for publication. Others, too, make the same complaint: the arguments involved are covered over such a huge mountain of posts that it's impossible for most outsiders to seriously evaluate them. This is a problem for both those who'd want to critique the concept, and for those who tentatively agree and would want to learn more about it.

Since it appears (do correct me if I'm wrong!) that Eliezer doesn't currently consider it worth the time and effort to do this, why not enlist the LW community in summarizing his arguments the best we can and submit them somewhere once we're done? Minds and Machines will be having a special issue on transhumanism, cognitive enhancement and AI, with a deadline for submission in January; that seems like a good opportunity for the paper. Their call for papers is asking for submissions that are around 4000 to 12 000 words.

The paper should probably

  • Briefly mention some of the previous work about AI being near enough to be worth consideration (Kurzweil, maybe Bostrom's paper on the subject, etc.), but not dwell on it; this is a paper on the consequences of AI.
  • Devote maybe little less than half of its actual content to the issue of FOOM, providing arguments and references for building the case of a hard takeoff.
  • Devote the second half to discussing the question of FAI, with references to e.g. Joshua Greene's thesis and other relevant sources for establishing this argument. Carl Shulman says SIAI is already working on a separate paper on this, so it'd be better for us to concentrate merely on the FOOM aspect.
  • Build on the content of Eliezer's various posts, taking their primary arguments and making them stronger by reference to various peer-reviewed work.
  • Include as authors everyone who made major contributions to it and wants to be mentioned; certainly make (again, assuming he doesn't object) Eliezer as the lead author, since this is his work we're seeking to convert into more accessible form.

I have created a wiki page for the draft version of the paper. Anyone's free to edit.

Comments (210)

Comment author: Daniel_Burfoot 17 November 2009 11:54:26PM *  11 points [-]

I hope such a document addresses the Mañana response: yeah, sure, figuring out how to control the AIs and make sure they're friendly is important, but there's no time pressure. It's not like powerful AI is going to be here anytime soon. In fact it's probably impossible to figure out how to control the AI, since we still have no idea how it will work.

I expect this kind of response is common among AI researchers, who do believe in the possibility of AI, but, having an up-close view of the sorry state of the field, have trouble getting excited about prophecies of doom.

Comment author: CronoDAS 19 November 2009 03:00:11AM *  1 point [-]

Yeah, sure, figuring out how to control the AIs and make sure they're friendly is important, but there's no time pressure. It's not like powerful AI is going to be here anytime soon.

This accurately describes my current position.

Comment author: MendelSchmiedekamp 20 November 2009 09:41:45PM 10 points [-]

It's as though no one here has ever heard of the bystander effect. The deadline is January 15th. Setting up a wiki page and saying "Anyone's free to edit." is the equivalent to killing this thing.

Also this is a philosophy, psychology, and technology journal, which means that despite the list of references for Singularity research you will also need to link this with the philosophical and/or public policy issues that the journal wants you to address (take a look at the two guest editors).

Another worry to me is that in all the back issues of this journal I looked over, the papers were almost always monographs (and baring that 2). I suspect that having many authors might kill the chances for this paper.

Comment author: Morendil 04 February 2010 08:47:30AM 2 points [-]

This prediction was right on the money.

This is being tracked on PredictionBook by the way. I have some reservations about the usefulness of PB in general, but one thing that is quite valuable is its providing a central "diary" of upcoming predictions made at various dates in the past, that would otherwise be easy to forget.

Comment author: Kaj_Sotala 20 November 2009 10:37:05PM 0 points [-]

The deadline is January 15th. Setting up a wiki page and saying "Anyone's free to edit." is the equivalent to killing this thing.

I know. I was hoping somebody'd take the initiative, but failing that I'll muster the time to actually contribute to the article at some point.

Comment author: Zack_M_Davis 20 November 2009 11:25:25PM 0 points [-]

Yeah, I'm having my doubts about the whole crowdsourcing thing, too.

I've started hacking away at the wiki page today; wanna be coauthors?

Comment author: MichaelAnissimov 17 November 2009 10:11:55PM 6 points [-]

The arguments that I found most compelling for a hard takeoff are found in LOGI part 3 and the Wiki interview with Eliezer from 2003 or so, for anyone who needs help on references or argument ideas from outside of the sequences.

Comment author: timtyler 17 November 2009 10:38:35PM 3 points [-]

"a point in time when the speed of technological progress becomes near-infinite (i.e., discontinuous), caused by advanced technologies"

"Near infinite" is mystical math. There's no such thing as "near infinite" in real maths. Things are either finite, or they are not.

Comment author: MichaelAnissimov 18 November 2009 01:57:32AM 1 point [-]

Yes Tim, that's absolutely correct. That alternate meaning is complete bullshit, but it exists nonetheless. Very unfortunate, but I see very few people taking an initiative towards stomping it out in the wider world.

Comment author: timtyler 18 November 2009 08:50:24AM *  7 points [-]

I think: "2) a point in time when prediction is no longer possible (a.k.a., "Predictive Horizon")" ...is equally nonsensical. Eliezer seems to agree:

"The Predictive Horizon never made much sense to me"

...and so does Nick, quoted later in the essay:

"I think it is unfortunate that some people have made Unpredictability a defining feature of "the singularity". It really does tend to create a mental block."

Robin Hanson thinks that the unpredictability idea is silly as well.

Yet aren't these two the main justifications for using the "singularity" term in the first place?

If the rate of progress is not about to shoot off to infinity, and there isn't going to be an event-horizon-like threshold at some future point in time, it seems to me that that's two of the major justifications for using the "singularity" term down the toilet.

To me - following the agricultural/industrial terminology - it looks as though there will be an intelligence revolution - and then probably a molecular nanotechnology/robotics revolution not long after.

Squishing those two concepts together into "singularity" paste offends my sense of the naming historical events. I think it is confusing, misleading, and pseudo-scientific.

Please quit with the ridiculous singularity terminology!

http://alife.co.uk/essays/the_singularity_is_nonsense/

Comment author: righteousreason 18 November 2009 05:33:14AM 0 points [-]

I thought similarly about LOGI part 3 (Seed AI). I actually thought of that immediately and put a link up to that on the wiki page.

Comment author: JamesAndrix 17 November 2009 09:51:59PM 6 points [-]

This is a problem for both those who'd want to critique the concept, and for those who are more open-minded and would want to learn more about it.

Nit: This implies that people who disagree are closed minded.

Comment author: Kaj_Sotala 18 November 2009 05:44:37PM 2 points [-]

Good point, that wasn't intentional. I'll edit it.

Comment author: Nick_Tarleton 20 November 2009 05:08:59PM *  5 points [-]

Devote the second half to discussing the question of FAI, with references to e.g. Joshua Greene's thesis and other relevant sources for establishing this argument.

(Now that this is struck out it might not matter, but) I wonder if, in addition to possibly overrating Greene's significance as an exponent of moral irrealism, we don't overrate the significance of moral realism as an obstacle to understanding FAI ("shiny pitchforks"). I would expect the academic target audience of this paper, especially the more technical subset, to be metaethically confused but not moral realists. Much more tentatively, I suspect that for the technical audience, more important than resolving metaethical confusion would be communicating the complexity of values: getting across "no, you really don't just value intelligence or knowledge or 'complexity', you really don't want humans replaced by arbitrary greater intelligences (or possibly replaced by anything; values can be path-dependent), and, while that would be true even if most superintelligences had complex values and built complex worlds, there's reason to think most would produce dull monocultures." ("Which Consequentialism? Machine Ethics and Moral Divergence" is closely related.)

But maybe I overstate the difference between that and metaethical confusion; both fall under "reasons to think we get a good outcome by default", both are supported by intuitions against "arbitrary" values, and probably have other psychology in common.

Comment author: RobinHanson 18 November 2009 02:25:31PM *  5 points [-]

A problem with this proposal is whether this paper can be seen as authorative. A critic might worry that if they study and respond to this paper they will be told it does not represent the best pro-Singularity arguments. So the paper would need to be endorsed enough to gain enough status to become worth criticizing.

Comment author: Arenamontanus 19 November 2009 06:25:14PM 6 points [-]

The way to an authoritative paper is not just to have the right co-authors but mainly having very good arguments, cover previous research well and ensure that it is out early in an emerging field. That way it will get cited and used. In fact, one strong reason to write this paper now is that if you don't do it, somebody else (and perhaps much worse) will do it.

Comment author: righteousreason 18 November 2009 10:46:07PM *  3 points [-]

Eliezer is arguing about one view of the Singularity, though there are others. This is one reason I thought to include http://yudkowsky.net/singularity/schools on the wiki. If leaders/proponents of the other two schools could acknowledge this model Eliezer has described of there being three schools of the Singularity, I think that might lend it more authority as you are describing.

Comment author: Kaj_Sotala 19 November 2009 08:29:33AM 4 points [-]

Actually, I might prefer not to use the term 'Singularity' at all, precisely because it has picked up so many different meanings. If a name is needed for the event we're describing and we can't avoid that, use 'intelligence explosion'.

Comment author: UnholySmoke 21 November 2009 12:26:53AM 2 points [-]

Seconded. One of the many modern connotations of 'Singularity' is 'Geek Apocalypse'.

Which is happening, like, a good couple of years afterwards.

Intelligence explosion does away with that, and seems to nail the concept much better anyway.

Comment author: Vladimir_Nesov 18 November 2009 02:42:33PM *  3 points [-]

In what contexts is the action you mention worth performing? Why are "critics" a relevant concern? In my perception, normal technical science doesn't progress by criticism, it works by improving on some of existing work and forgetting the rest. New developments allow to see some old publications as uninteresting or wrong.

Comment author: mormon2 18 November 2009 04:46:59PM 3 points [-]

"In what contexts is the action you mention worth performing?"

If the paper was endorsed by the top minds who support the singularity. Ideally if it was written by them. So for example Ray Kurzweil whether you agree with him or not he is a big voice for the singularity.

"Why are "critics" a relevant concern?"

Because technical science moves forward through peer-review and the proving and the disproving of hypotheses. The critics help prevent the circle jerk phenomena in science assuming they are well thought out critiques. Because outside review can sometimes see fatal flaws in ideas that are not necessarily caught by those who work in the field.

"In my perception, normal technical science doesn't progress by criticism, it works by improving on some of existing work and forgetting the rest. New developments allow to see some old publications as uninteresting or wrong."

Have you ever published in a peer-review journal? If not the last portion of your post I will ignore, if so perhaps your could expound on it a bit more.

Comment author: Vladimir_Nesov 18 November 2009 05:16:20PM 1 point [-]

Have you ever published in a peer-review journal? If not the last portion of your post I will ignore, if so perhaps your could expound on it a bit more.

The actual experience of publishing a paper hardly adds anything that can't be understood without doing so. Peer-review is not about "critics" responding to endorsement by well-known figures, it's quality control (with whatever failing it may carry), and not a point where written-up public criticisms originate. Science builds on what's published, not on what gets rejected by peer review, and what's published can be read by all.

Comment author: Tyrrell_McAllister 19 November 2009 04:43:56PM 1 point [-]

FWIW, in my experience the useful criticisms happen at conferences or in private conversation, not during the peer review process.

Comment author: MichaelVassar 19 November 2009 03:54:42PM -1 points [-]

It is rarely the case that experience adds hardly anything. What are your priors and posteriors here? How did you update?

Comment author: wedrifid 18 November 2009 11:04:58PM 1 point [-]

A problem with this proposal is whether this paper can be seen as authorative. A critic might worry that if they study and respond to this paper they will be told it does not represent the best pro-Singularity arguments. So the paper would need to be endorsed enough to gain enough status to become worth criticizing.

This is the main reasons Eliezer gives as to why he does not bother to create such a proposal. If this does apply to Eliezer as well would a suggestion that he should write such a paper serve any other purpose than one-upmanship?

Comment author: Risto_Saarelma 18 November 2009 06:06:27AM 5 points [-]

One argument against a hard takeoff is Greg Egan's idea about there not being a strong hierarchy of power in general intelligence, which he goes into here, for example. Would this be worth covering?

Comment author: Kaj_Sotala 18 November 2009 05:43:29PM 0 points [-]

Definitely so, at least briefly.

Comment author: timtyler 24 November 2009 10:17:40PM -1 points [-]

I don't think that's worth very much. We already have systems which are much more powerful in many areas.

Comment author: Arenamontanus 19 November 2009 06:36:18PM 4 points [-]

I think this is very needed. When reviewing singularity models for a paper I wrote I could not find many readily citable references to certain areas that I know exist as "folklore". I don't like mentioning such ideas because it makes it look (to outsiders) as I have come up with them, and the insiders would likely think I was trying to steal credit.

There are whole fields like friendly AI theory that need a big review. Both to actually gather what has been understood, and in order to make it accessible to outsiders so that the community thinking about it can grow and deepen.

Whether this is a crowdsourcable project is another matter, but at the very least crowdsourcing raw input for later expert paper construction sounds like a good idea. I would expect that eventually it would need to boil down to one or two main authors doing most of the job, and a set of co-authors for speciality skills and prestige. But since this community is less driven by publish-or-perish and more by rationality concerns I expect ordering of co-authors may be less important.

Comment author: kip1981 18 November 2009 01:19:22AM 4 points [-]

I would be surprised if Eliezer would cite Joshua Greene's moral anti-realist view with approval.

Comment author: Eliezer_Yudkowsky 18 November 2009 02:24:24AM 5 points [-]

Correct. I'm a moral cognitivist; "should" statements have truth-conditions. It's just that very few possible minds care whether should-statements are true or not; most possible minds care about whether alien statements (like "leads-to-maximum-paperclips") are true or not. They would agree with us on what should be done; they just wouldn't care, because they aren't built to do what they should. They would similarly agree with us that their morals are pointless, but would be concerned with whether their morals are justified-by-paperclip-production, not whether their morals are pointless. And under ordinary circumstances, of course, they would never formulate - let alone bother to compute - the function we name "should" (or the closely related functions "justifiable" or "arbitrary").

Comment author: RobinZ 18 November 2009 10:12:30PM *  4 points [-]

Am I right in interpreting the bulk of the thread following this comment (excepting perhaps the FAI derail) as a dispute on the definition of "should"?

Comment author: CronoDAS 19 November 2009 02:58:05AM 2 points [-]

Yes, we're disputing definitions, which tends to become pointless. However, we can't seem to get away from using the word "should", so we might as well get it pinned down to something we can agree upon.

Comment author: wedrifid 18 November 2009 10:52:52PM *  0 points [-]

Am I right in interpreting the bulk of the thread following this comment (excepting perhaps the FAI derail) as a dispute on the definition of "should"?

I think you are right.

The dispute also serves as a signal of what some parts of the disputants personal morality probably includes. This is fitting with the practical purpose that the concept 'should' has in general. Given what Eliezer has chosen as his mission this kind of signalling is a matter of life and death in the same way that it would have been in our environment of evolutionary adaptation. That is, if people had sufficient objection to Eliezer's values they would kill him rather than let him complete an AI.

Comment author: CronoDAS 18 November 2009 07:42:46AM *  4 points [-]

Here's a very short unraveling of "should":

"Should" means "is such so as to fulfill the desires in question." For example, "If you want to avoid being identified while robbing a convenience store, you should wear a mask."

In the context of morality, the desires in question are all desires that exist. "You shouldn't rob convenience stores" means, roughly, "People in general have many and strong reasons to ensure that individuals don't want to rob convenience stores."

For the long version, see http://atheistethicist.blogspot.com/2005/12/meaning-of-ought.html .

Comment author: wuwei 21 November 2009 01:29:15AM *  3 points [-]

I'm a moral cognitivist too but I'm becoming quite puzzled as to what truth-conditions you think "should" statements have. Maybe it would help if you said which of these you think are true statements.

1) Eliezer Yudkowsky should not kill babies.

2) Babyeating aliens should not kill babies.

3) Sharks should not kill babies.

4) Volcanoes should not kill babies.

5) Should not kill babies. (sic)

The meaning of "should not" in 2 through 5 are intended to be the same as the common usage of the words in 1.

Comment author: Clippy 21 November 2009 01:38:50AM 10 points [-]

Technically, you would need to include a caveat in all of those like, "unless to do so would advance paperclip production" but I assume that's what you meant.

Comment author: Nick_Tarleton 21 November 2009 02:12:09AM *  2 points [-]

The meaning of "should not" in 2 through 5 are intended to be the same as the common usage of the words in 1.

I don't think there is one common usage of the word "should".

(ETA: I asked the nearest three people if "volcanoes shouldn't kill people" is true, false, or neither, assuming that "people shouldn't kill people" is true or false so moral non-realism wasn't an issue. One said true, two said neither.)

Comment author: Eliezer_Yudkowsky 21 November 2009 01:34:01AM 0 points [-]

They all sound true to me.

Comment author: wuwei 21 November 2009 03:27:57AM 1 point [-]

Interesting, what about either of the following:

A) If X should do A, then it is rational for X to do A.

B) If it is rational for X to do A, then X should do A.

Comment author: wedrifid 21 November 2009 04:42:53AM *  1 point [-]

From what I understand of what Eliezer's position:

A) If X should do A, then it is rational for X to do A.

False

B) If it is rational for X to do A, then X should do A.

False.

(If this isn't the case then Eliezer's 'should' is even more annoying than how I now understand it.)

Comment author: Eliezer_Yudkowsky 21 November 2009 08:57:27AM 0 points [-]

Yep, both false.

Comment author: komponisto 22 November 2009 12:57:05PM 1 point [-]

So, just to dwell on this for a moment, there exist X and A such that (1) it is rational for X to do A and (2) X should not do A.

How do you reconcile this with "rationalists should win"? (I think I know what your response will be, but I want to make sure.)

Comment author: arundelo 22 November 2009 04:17:36PM 1 point [-]

Here's my guess at one type of situation Eliezer might be thinking of when calling proposition B false: It is rational (let us stipulate) for a paperclip maximizer to turn all the matter in the solar system into computronium in order to compute ways to maximize paperclips, but "should" does not apply to paperclip maximizers.

Comment author: Eliezer_Yudkowsky 22 November 2009 06:02:15PM -2 points [-]

Correct.

EDIT: If I were picking nits, I would say, "'Should' does apply to paperclip maximizers - it is rational for X to make paperclips but it should not do so - however, paperclip maximizers don't care and so it is pointless to talk about what they should do." But the overall intent of the statement is correct - I disagree with its intent in neither anticipation nor morals - and in such cases I usually just say "Correct". In this case I suppose that wasn't the best policy, but it is my usual policy.

Comment author: Stuart_Armstrong 18 November 2009 12:26:09PM *  3 points [-]

the function we name "should"

What I think you mean is:

There is a function Should(human) (or Should(Eliezer)) which computes the human consensus (or Eliezer's opinion) on what the morally correct course of action is.

And some alien beliefs have their own Should function which would be, in form if not in content, similar to our own. So a paperclip maximiser doesn't get a should, as it simply follows a "figure out how to maximise paper clips - then do it" format. However a complex alien society that has many values and feels they must kill everyone else for the artistic cohesion of the universe, but often fails to act on this feeling because of akrasia, will get a Should(Krikkit) function.

However, until such time as we meet this alien civilization, we should just use Should as a shorthand for Should(human).

Is my understanding correct?

Comment author: Eliezer_Yudkowsky 18 November 2009 07:06:39PM *  -2 points [-]

There could be a word defined that way, but for purposes of staying unconfused about morality, I prefer to use "would-want" so that "should" is reserved specifically for things that, you know, actually ought to be done.

Comment author: timtyler 21 November 2009 08:46:29AM 2 points [-]

"would-want" - under what circumstances? Superficially, it seems like pointless jargon. Is there a description somewhere of what it is supposed to mean?

Comment author: timtyler 24 November 2009 10:29:48PM -1 points [-]

Hmm. I guess not.

Comment author: Stuart_Armstrong 18 November 2009 07:14:50PM 1 point [-]

Fair enough. But are you saying that there is an objective standard of ought, or do you just mean a shared subjective standard? Or maybe a single subjective standard?

Comment author: Eliezer_Yudkowsky 18 November 2009 07:28:57PM *  14 points [-]

The word "ought" means a particular thing, refers to a particular function, and once you realize that, ought-statements have truth-values. There's just nothing which says that other minds necessarily care about them. It is also possible that different humans care about different things, but there's enough overlap that it makes sense (I believe, Greene does not) to use words like "ought" in daily communication.

What would the universe look like if there were such a thing as an "objective standard"? If you can't tell me what the universe looks like in this case, then the statement "there is an objective morality" is not false - it's not that there's a closet which is supposed to contain an objective morality, and we looked inside it, and the closet is empty - but rather the statement fails to have a truth-condition. Sort of like opening a suitcase that actually does contain a million dollars, and you say "But I want an objective million dollars", and you can't say what the universe would look like if the million dollars were objective or not.

I should write a post at some point about how we should learn to be content with happiness instead of "true happiness", truth instead of "ultimate truth", purpose instead of "transcendental purpose", and morality instead of "objective morality". It's not that we can't obtain these other things and so must be satisfied with what we have, but rather that tacking on an impressive adjective results in an impressive phrase that fails to mean anything. It is not that there is no ultimate truth, but rather, that there is no closet which might contain or fail to contain "ultimate truth", it's just the word "truth" with the sonorous-sounding adjective "ultimate" tacked on in front. Truth is all there is or coherently could be.

Comment author: dclayh 19 November 2009 08:16:40PM 8 points [-]

I should write a post at some point about how we should learn to be content with happiness instead of "true happiness", truth instead of "ultimate truth", purpose instead of "transcendental purpose", and morality instead of "objective morality".

When you put those together like that it occurs to me that they all share the feature of being provably final. I.e., when you have true happiness you can stop working on happiness; when you have ultimate truth you can stop looking for truth; when you know an objective morality you can stop thinking about morality. So humans are always striving to end striving.

(Of course whether they'd be happy if they actually ended striving is a different question, and one you've written eloquently about in the "fun theory" series.)

Comment author: Eliezer_Yudkowsky 19 November 2009 08:35:52PM 3 points [-]

That's actually an excellent way of thinking about it - perhaps the terms are not as meaningless as I thought.

Comment author: Stuart_Armstrong 20 November 2009 04:17:57PM 5 points [-]

The word "ought" means a particular thing, refers to a particular function, and once you realize that, ought-statements have truth-values. There's just nothing which says that other minds necessarily care about them. It is also possible that different humans care about different things, but there's enough overlap that it makes sense (I believe, Greene does not) to use words like "ought" in daily communication.

Just a minor thought: there is a great deal of overlap on human "ought"s, but not so much on formal philosphical "ought"s. Dealing with philosophers often, I prefer to see ought as a function, so I can talk of "ought(Kantian)" and "ought(utilitarian)".

Maybe Greene has more encounters with formal philosophers than you, and thus cannot see much overlap?

Comment author: timtyler 18 November 2009 07:51:29PM *  5 points [-]

Re: "The word "ought" means a particular thing, refers to a particular function, and once you realize that, ought-statements have truth-values."

A reveling and amazing comment - from my point of view. I had no idea you believed that.

What about alien "ought"s? Presumably you can hack the idea that aliens might see morality rather differently from us. So, presumably you are talking about ought<human> - glossing over our differences from one another.

There's a human morality in about the same sense as there's a human height.

Comment author: Eliezer_Yudkowsky 18 November 2009 08:54:56PM *  3 points [-]

What about alien "ought"s?

There are no alien oughts, though there are alien desires and alien would-wants. They don't see morality differently from us; the criterion by which they choose is simply not that which we name morality.

There's a human morality in about the same sense as there's a human height.

This is a wonderful epigram, though it might be too optimistic. The far more pessimistic version would be "There's a human morality in about the same sense as there's a human language." (This is what Greene seems to believe and it's a dispute of fact.)

Comment author: Wei_Dai 18 November 2009 10:38:37PM *  5 points [-]

Eliezer, I think your proposed semantics of "ought" is confusing, and doesn't match up very well with ordinary usage. May I suggest the following alternative?

ought<X> refer's to X's would-wants if X is an individual. If X is a group, then ought<X> is the overlap between the oughts of its members.

In ordinary conversation, when people use "ought" without an explicit subscript or possessive, the implicit X is the speaker plus the intended audience (not humanity as a whole).

ETA: The reason we use "ought" is to convince the audience to do or not do something, right? Why would we want to refer to ought<humanity>, when ought<speaker+audience> would work just fine for that purpose, and ought<speaker+audience> covers a lot more ground than ought<humanity>?

Comment author: wedrifid 19 November 2009 06:43:47PM 3 points [-]

"There's a human morality in about the same sense as there's a human language." (This is what Greene seems to believe and it's a dispute of fact.)

That seems to hit close to the mark. Human language contains all sorts of features that are more or less universal to humans due to their hardware while also being significantly determined by cultural influences. It also shares the feature that certain types of language (and 'ought' systems) are more useful in different cultures or subcultures.

This is a wonderful epigram, though it might be too optimistic. The far more pessimistic version would be

I'm not sure I follow this. Neither seem particularly pessimistic to me and I'm not sure how one could be worse than the other.

Comment author: timtyler 18 November 2009 09:55:43PM 1 point [-]

"There are no alien oughts" and "They don't see morality differently from us" - these seem like more bizarre-sounding views on the subject of morality - and it seems especially curious to hear them from the author of the "Baby-Eating Aliens" story.

Comment author: Furcas 18 November 2009 10:14:27PM 5 points [-]

Look, it's not very complicated: When you see Eliezer write "morality" or "oughts", read it as "human morality" and "human oughts".

Comment author: RobinZ 19 November 2009 01:11:42AM 1 point [-]

Jumping recklessly in at the middle: even granting your premises regarding the scope of 'ought', it is not wholly clear that an alien "ought" is impossible. As timtyler pointed out, the Babyeaters in "Three Worlds Collide" probably had a would-want structure within the "ought" cluster in thingspace, and systems of behaviors have been observed in some nonhuman animals which resemble human morality.

I'm not saying it's likely, though, so this probably constitutes nitpicking.

Comment author: NancyLebovitz 19 November 2009 02:06:32PM 2 points [-]

Then we need a better way of distinguishing between what we're doing and what we would be doing if we were better at it.

You've written about the difference between rationality and believing that one's bad arguments are rational.

For the person who is in the latter state, something that might be called "true rationality" is unimaginable, but it exists.

Comment author: Stuart_Armstrong 19 November 2009 08:16:29AM 2 points [-]

Thanks, this has made your position clear. And - apart from tiny differences in vocabulary - it is exactly the same as mine.

Comment author: Kaj_Sotala 18 November 2009 07:53:43PM 1 point [-]

It is not that there is no ultimate truth, but rather, that there is no closet which might contain or fail to contain "ultimate truth", it's just the word "truth" with the sonorous-sounding adjective "ultimate" tacked on in front. Truth is all there is or coherently could be.

So what about the Ultimate Showdown of Ultimate Destiny?

...sorry, couldn't resist.

Comment author: Eliezer_Yudkowsky 18 November 2009 08:08:24PM 1 point [-]

But there is a truth-condition for whether a showdown is "ultimate" or not.

Comment author: Zack_M_Davis 18 November 2009 08:10:01PM -1 points [-]

It is [...] possible that different humans care about different things, but there's enough overlap that it makes sense (I believe, Greene does not) to use words like "ought" in daily communication.

This sentence is much clearer than the sort of thing you usually say.

Comment author: wedrifid 21 November 2009 10:36:42AM 1 point [-]

Fair enough. But are you saying that there is an objective standard of ought, or do you just mean a shared subjective standard? Or maybe a single subjective standard?

A single subjective standard. But he uses different terminology, with that difference having implications about how morality should (full Eliezer meaning) be thought about.

It can be superficially considered to be a shared subjective standard in as much as many other humans have morality that overlaps with his in some ways and also in the sense that his morality includes (if I recall correctly) the preferences of others somewhere within it. I find it curious that the final result leaves language and positions that are reminiscent of those begot by a belief in an objective standard of ought but without requiring totally insane beliefs like, say, theism or predicting that a uFAI will learn 'compassion' and become a FAI just because 'should' is embedded in the universe as an inevitable force or something.

Still, if I am to translate the Eliezer word into the language of Stuart_Armstrong it matches "a single subjective standard but I'm really serious about it". (Part of me wonders if Eliezer's position on this particular branch of semantics would be any different if there were less non-sequitur rejections of Bayesian statistics with that pesky 'subjective' word in it.)

Comment author: SforSingularity 18 November 2009 02:52:30AM 1 point [-]

Correct. I'm a moral cognitivist;

I think you're just using different words to say the same thing that Greene is saying, you in particular use "should" and "morally right" in a nonstandard way - but I don't really care about the particular way you formulate the correct position, just as I wouldn't care if you used the variable "x" where Greene used "y" in an integral.

You do agree that you and Greene are actually saying the same thing, yes?

Comment author: Eliezer_Yudkowsky 18 November 2009 02:58:48AM 2 points [-]

You do agree that you and Greene are actually saying the same thing, yes?

I don't think we anticipate different experimental results. We do, however, seem to think that people should do different things.

Comment author: SforSingularity 18 November 2009 04:29:20AM *  3 points [-]

people should do different things.

Whose version of "should" are you using in that sentence? If you're using the EY version of "should" then it is not possible for you and Greene to think people should do different things unless you and Greene anticipate different experimental results...

... since the EY version of "should" is (correct me if I am wrong) a long list of specific constraints and valuators that together define one specific utility function U _ humanmoralityaccordingtoEY. You can't disagree with Greene over what the concrete result of maximizing U _ humanmoralityaccordingtoEY is unless one of you is factually wrong.

Comment author: Eliezer_Yudkowsky 18 November 2009 04:33:05AM 3 points [-]

Oh well in that case, we disagree about what reply we would hear if we asked a friendly AI how to talk and think about morality in order to maximize human welfare as construed in most traditional utilitarian senses.

This is phrased as a different observable, but it represents more of a disagreement about impossible possible worlds than possible worlds - we disagree about statements with truth conditions of the type of mathematical truth, i.e. which conclusions are implied by which premises. Though we may also have some degree of empirical disagreement about what sort of talk and thought leads to which personal-hedonic results and which interpersonal-political results.

(It's a good and clever question, though!)

Comment author: SforSingularity 18 November 2009 04:52:41AM 3 points [-]

we disagree about what reply we would hear if we asked a friendly AI how to talk and think about morality in order to maximize human welfare as construed in most traditional utilitarian senses.

Surely you should both have large error bars around the answer to that question in the form of fairly wide probability distributions over the set of possible answers. If you're both well-calibrated rationalists those distributions should overlap a lot. Perhaps you should go talk to Greene? I vote for a bloggingheads.

Comment author: wedrifid 18 November 2009 05:19:28AM 0 points [-]

I vote for a bloggingheads.

Wouldn't that be 'advocate', 'propose' or 'suggest'?

Comment author: bgrah449 18 November 2009 07:38:07PM 1 point [-]

I vote no, it wouldn't be

Comment author: Eliezer_Yudkowsky 18 November 2009 05:11:34AM 1 point [-]

Asked Greene, he was busy.

Yes, it's possible that Greene is correct about what humanity ought to do at this point, but I think I know a bit more about his arguments than he does about mine...

Comment author: SforSingularity 18 November 2009 06:15:44AM 1 point [-]

That is plausible.

Comment author: wuwei 18 November 2009 04:20:42AM 0 points [-]

I don't think we anticipate different experimental results.

I find that quite surprising to hear. Wouldn't disagreements about meaning generally cash out in some sort of difference in experimental results?

Comment author: wuwei 18 November 2009 04:14:08AM *  1 point [-]

On your analysis of should, paperclip maximizers should not maximize paperclips. Do you think this is a more useful characterization of 'should' than one in which we should be moral and rational, etc., and paperclip maximizers should maximize paperclips?

Comment author: Eliezer_Yudkowsky 18 November 2009 04:27:14AM 2 points [-]

A paperclip maximizer will maximize paperclips. I am unable to distinguish any sense in which this is a good thing. Why should I use the word "should" to describe this, when "will" serves exactly as well?

Comment author: Vladimir_Nesov 18 November 2009 11:43:49AM *  4 points [-]

Please amplify on that. I can sorta guess what you mean, but can't be sure.

We make a distinction between the concepts of what people will do and what they should do. Is there an analogous pair of concepts applicable to paperclip maximizers? Why or why not? If not, what is the difference between people and paperclip maximizers that justifies there being this difference for people but not for paperclip maximizers?

A paperclip maximizer will maximize paperclips.

Will paperclip maximizers, when talking about themselves, distinguish between what they will do, and what will maximize paperclips? (While wishing they'd be more paperclip maximizers they wish they were.) What they will actually do is distinct from what will maximize paperclips: it's predictable that actual performance is always less than optimal, given the problem is open-ended enough.

Comment author: Eliezer_Yudkowsky 18 November 2009 07:09:15PM *  6 points [-]

Let there be a mildly insane (after the fashion of a human) paperclipper named Clippy.

Clippy does A. Clippy would do B if a sane but bounded rationalist, C if an unbounded rationalist, and D if it had perfect veridical knowledge. That is, D is the actual paperclip-maximizing action, C is theoretically optimal given all of Clippy's knowledge, B is as optimal as C can realistically get under perfect conditions.

Is B, C, or D what Clippy Should(Clippy) do? This is a reason to prefer "would-want". Though I suppose a similar question applies to humans. Still, what Clippy should do is give up paperclips and become an FAI. There's no chance of arguing Clippy into that, because Clippy doesn't respond to what we consider a moral argument. So what's the point of talking about what Clippy should do, since Clippy's not going to do it? (Nor is it going to do B, C, or D, just A.)

PS: I'm also happy to talk about what it is rational for Clippy to do, referring to B.

Comment author: Vladimir_Nesov 18 November 2009 07:31:45PM 3 points [-]

Your usage of 'should' is more of a redefinition than clarification. B,C and D work as clarifications for the usual sense of the word: "should" has a feel 'meta' enough to transfer over to more kinds of agents.

If you can equally well talk of Should(Clippy) and Should(Humanity), then for the purposes of FAI it's Should that needs to be understood, not one particular sense should=Should(Humanity). If one can't explicitly write out Should(Humanity), one should probably write out Should(-), which is featureless enough for there to be no problem with the load of detailed human values, and in some sense pass Humanity as a parameter to its implementation. Do you see this framing as adequate or do you know of some problem with it?

Comment author: Eliezer_Yudkowsky 18 November 2009 07:38:28PM *  2 points [-]

This is a good framing for explaining the problem - you would not, in fact, try to build the same FAI for Clippies and humans, and then pass it humans as a parameter.

E.g. structural complications of human "should" that only the human FAI would have to be structurally capable of learning. (No, you cannot have complete structural freedom because then you cannot do induction.)

Comment author: Vladimir_Nesov 18 November 2009 09:34:00PM *  1 point [-]

This is a good framing for explaining the problem - you would not, in fact, try to build the same FAI for Clippies and humans, and then pass it humans as a parameter.

I expect you would build the same FAI for paperclipping (although we don't have any Clippies to pass it as parameter), so I'd appreciate it if you did explain the problem given you believe there is one, since it's a direction that I'm currently working.

Humans are stuff, just like any other feature of the world, that FAI would optimize, and on stuff-level it makes no difference that people prefer to be "free to optimize". You are "free to optimize" in a deterministic universe, it's the way this stuff is (being) arranged that makes the difference, and it's the content of human preference that says it shouldn't have some features like undeserved million-dollar bags falling from the sky, where undeserved is another function of stuff. An important subtlety of preference is that it makes different features of perhaps mutually exclusive possible scenarios depend on each other, so the fact that one should care about what could be and how it's related to what could be otherwise and even to how it's chosen what to actually realize is about scope of what preference describes, not about specific instance of preference. That is, in a manner of speaking, it's saying that you need an Int32, not a Bool to hold this variable, but that Int32 seems big enough.

Furthermore, considering the kind of dependence you described in that post you linked seems fundamental from a certain logical standpoint, for any system (not even "AI"). If you build the ontology for FAI on its epistemology, that is you don't consider it as already knowing anything but only as having its program that could interact with anything, then the possible futures and its own decision-making are already there (and it's all there is, from its point of view). All it can do, on this conceptual level, is to craft proofs (plans, designs of actions) that have the property of having certain internal dependencies in them, with the AI itself being the "current snapshot" of what it's planning. That's enough to handle the "free to optimize" requirement, given the right program.

Hmm, I'm essentially arguing that universal-enough FAI is "computable", that there is a program that computes a FAI for any given "creature", within a certain class of "creatures". I guess this problem is void, since obviously on the too-big-class side, for a small enough class this problem is in principle solvable, and for a big enough class it'll hit problems, if not conceptual then practical.

So the real question is about the characteristics of such class of systems for which it's easier to build an abstract FAI, that is a tool that takes a specimen of this class as a parameter and becomes a custom-made FAI for that specimen. This class needs to at least include humanity, and given the size of humanity's values, it needs to also include a lot of other stuff, for itself to be small enough to program explicitly. I currently expect a class of parameters of a manageable abstract FAI implementation to include even rocks and trees, since I don't see how to rigorously define and use in FAI theory the difference between these systems and us.

This also takes care of human values/humanity's values divide: these are just different systems to parameterize the FAI with, so there is no need for a theory of "value overlaps" distinct from a theory of "systems values". Another question is that "humanity" will probably be a bit harder to specify as parameter than some specific human or group of people.

Comment author: timtyler 21 November 2009 08:40:14AM 0 points [-]

Re: I suppose a similar question applies to humans.

Indeed - this objection is the same for any agent, including humans.

It doesn't seem to follow that the "should" term is inappropriate. If this is a reason for objecting to the "should" term, then the same argument concludes that it should not be used in a human context either.

Comment author: wedrifid 18 November 2009 05:55:58AM 1 point [-]

Why should I use the word "should" to describe this, when "will" serves exactly as well?

'Will' does not serve exactly as well when considering agents with limited optimisation power (that is, any actual agent). Considering, for example, a Paperclip Maximiser that happens to be less intelligent than I am. I may be able to predict that Clippy will colonize Mars before he invades earth but also be quite sure that more paperclips would be formed if Clippy invaded Earth first. In this case I will likely want a word that means "would better serve to maximise the agent's expected utility even if the agent does not end up doing it".

One option is to take 'should' and make it the generic 'should<Agent>'. I'm not saying you should use 'should' (implicitly, 'should<Clippy>') to describe the action that Clippy would take if he had sufficient optimisation power. But I am saying that 'will' does not serve exactly as well.

Comment author: Eliezer_Yudkowsky 18 November 2009 07:15:26AM 1 point [-]

I use "would-want" to indicate extrapolation. I.e., A wants X but would-want Y. This helps to indicate the implicit sensitivity to the exact extrapolation method, and that A does not actually represent a desire for Y at the current moment, etc. Similarly, A does X but would-do Y, A chooses X but would-choose Y, etc.

Comment author: timtyler 18 November 2009 12:24:42PM -1 points [-]

"Should" is a standard word for indicating moral obligation - it seems only sensible to use it in the context of other moral systems.

Comment author: timtyler 18 November 2009 12:32:41PM *  -1 points [-]

It's a good thing - from their point of view. They probably think that there should be more paperclips. The term "should" makes sense in the context of a set of preferences.

Comment author: Eliezer_Yudkowsky 18 November 2009 07:05:35PM *  1 point [-]

No, it's a paperclip-maximizing thing. From their point of view, and ours. No disagreement. They just care about what's paperclip-maximizing, not what's good.

Comment author: timtyler 18 November 2009 07:43:36PM *  0 points [-]

This is not a real point of disagreement.

IMO, in this context, "good" just means "favoured by this moral system". An action that "should" be performed is just one that would be morally obligatory - according to the specified moral system. Both terms are relative to a set of moral standards.

I was talking as though a paperclip maximiser would have morals that reflected their values. You were apparently assuming the opposite. Which perspective is better would depend on which particular paperclip maximiser was being examined.

Personally, I think there are often good reasons for morals and values being in tune with one another.

Comment author: AndrewKemendo 18 November 2009 02:32:11AM *  3 points [-]

This is a problem for both those who'd want to critique the concept, and for those who are more open-minded and would want to learn more about it.

Anyone who is sufficiently technically minded undoubtedly finds frustration in reading books which give broad brush stroked counterfactuals to decision making and explanation without delving into the details of their processes. I am thinking of books like Freakonomics, Paradox of Choice, Outliers, Nudge etc..

These books are very accessible but lack the in depth analysis which are expected to be thoroughly critiqued and understood in depth. Writings like Global catastrophic risks and any of the other written deconstructions of the necessary steps of technological singularity lack those spell-it-out-for-us-all sections that Gladwell et al. make their living from. Reasonably so. The issue of singularity is so much more complex and involved that it does not do the field justice to give slogans and banner phrases. Indeed it is arguably detrimental and has the ability to backfire by simplifying too much.

I think however what is needed is a clear, short and easily understood consensus on why this crazy AI thing is the inevitable result of reason, why it is necessary to think about, how it will help humanity, how it could reasonably hurt humanity.

The SIAI tried to do this:

http://www.singinst.org/overview/whatisthesingularity

http://www.singinst.org/overview/whyworktowardthesingularity

Neither of these is compelling in my view. They both go into some detail and leave the un-knowledgeable reader behind. Most importantly neither has what people want: a clear vision of exactly what we are working for. The problem is there isn't a clear vision; there is no consensus on how to start. Which is why in my view the SIAI is more focused on "Global risks" rather than just stating "We want to build an AI"; frankly, people get scared by the latter.

So is this paper going to resolve the dichotomy between the simplified and complex approach, or will we simply be replicating what the SIAI has already done?

Comment author: righteousreason 18 November 2009 05:35:40AM 3 points [-]

I found the two SIAI introductory pages very compelling the first time I read them. This was back before I knew what SIAI or the Singularity really was, as soon as I read through those I just had to find out more.

Comment author: CarlShulman 17 November 2009 08:19:36PM *  6 points [-]

This is a great idea Kaj, thanks for taking the initiative.

As noted by others, one issue with the AI Risks chapter is that it attempts to cover so much ground. I would suggest starting with just hard take-off, or local take-off, and presenting a focused case for that, without also getting into the FAI questions. This could also cut back on some duplication of effort, as SIAI folk were already planning to submit a paper (refined from some work done for a recent conference) for that issue on "machine ethics for superintelligence", which will be discussing the FAI problem.

P.S. If people would like to volunteer to review drafts of those papers in a few weeks to give feedback before they are submitted, that would be much appreciated. (You can email me at myfirstname AT mylastname at gmail)

Comment author: wedrifid 18 November 2009 01:57:10AM 2 points [-]

myfirstname AT mylastname at gmail

Is the first AT to really confuse bots or am I missing something technical?

Comment author: Zack_M_Davis 18 November 2009 02:48:23AM *  1 point [-]

I think it was supposed to be a "dahtt."

Comment author: Kaj_Sotala 18 November 2009 05:44:20PM 0 points [-]

Thanks for the heads-up - if there's already a FAI paper in the works, then yes, it's certainly better for this one to concentrate merely on the FOOM aspect.

Comment author: whpearson 17 November 2009 03:33:11PM 2 points [-]

As a Foom skeptic, what would convince me of taking the concept seriously, is an argument that intelligence/power is a quantity that we reason with in the same way as we reason about the number of neutrons in a nuclear reactor/bomb. Power seems like a slippery ephemeral concept, optimisation power appears to be able evaporate at the drop of a hat (if someone comes to know an opponents source code and can emulate them entirely).

Comment author: Johnicholas 17 November 2009 04:21:36PM 3 points [-]

I am also to some extent skeptical but I can recreate some of the arguments for the possibility of AI-caused explosive change that I've heard.

  1. From the timescale of nonliving matter, life seemed explosive.
  2. From the timescale of nonhuman life, humans seemed explosive.
  3. If we look to Moore's law (or other similar metrics) of the degree of processing power (or of knowledge) "possessed" by human-crafted artifacts, there seems to be an explosive trend.

We certainly take the concept of intelligence and intellectual power seriously when we deal with other humans (e.g. employers evaluating potential hires, or in diagnosing mental retardation). To some extent, we expect that a human with computational tools (abacus, pen and paper, calculator, computer) will have increased intellectual capabilities - so we do judge the power of "cyborg" human-computer amalgams.

Comment author: timtyler 17 November 2009 10:44:41PM 1 point [-]

We have explosive change today - if "explosive" intended to mean the type of exponential growth process exhibited in nuclear bombs. Check with Moore's law.

If you are looking for an explosion, there is no need for a crystal ball - simply look around you.

Comment author: Johnicholas 18 November 2009 03:09:09AM 0 points [-]

I agree with you - and I think the SIAI focuses too much on possible future computer programs, and neglects the (limited) superintelligences that already exist, various amalgams and cyborgs and group minds coordinated with sonic telepathy.

In the future where the world continues (that is, without being paperclipped) and without a singleton, we need to think about how to deal with superintelligences. By "deal with" I'm including prevailing over superintelligences, without throwing up hands and saying "it's smarter than me".

Comment author: CronoDAS 18 November 2009 06:26:51AM 3 points [-]

By "deal with" I'm including prevailing over superintelligences, without throwing up hands and saying "it's smarter than me".

You can start practicing by trying to beat your computer at chess. ;)

Comment author: Johnicholas 18 November 2009 07:20:33PM *  2 points [-]

I'm pretty good at beating my computer at chess, even though I'm an awful player. I challenge it, and it runs out of time - apparently it can't tell that it's in a competition, or can't press the button on the clock.

This might sound like a facetious answer, but I'm serious. One way to defeat something that is stronger than you in a limited domain is to strive to shift the domain to one where you are strong. Operating with objects designed for humans (like physical chess boards and chess clocks) is a domain that current computers are very weak at.

There are other techniques too. Consider disease-fighting. The microbes that we fight are vastly more experienced (in number of generations evolved), and the number of different strategies that they try is vastly huge. How is it that we manage to (sometimes) defeat specific diseases? We strive to hamper the enemy's communication and learning capabilities with quarantine techniques, and steal or copy the nanotechnology (antibiotics) necessary to defeat it. These strategies might well be our best techniques against unFriendly manmade nanotechnological infections, if such broke out tomorrow.

Bruce Schneier beats people over the head with the notion DON'T DEFEND AGAINST MOVIE PLOTS! The "AI takes over the world" plot is influencing a lot of people's thinking. Unfriendly AGI, despite its potential power, may well have huge blind spots; mind design space is big!

Comment author: wedrifid 21 November 2009 06:04:04AM *  5 points [-]

Bruce Schneier beats people over the head with the notion DON'T DEFEND AGAINST MOVIE PLOTS! The "AI takes over the world" plot is influencing a lot of people's thinking. Unfriendly AGI, despite its potential power, may well have huge blind spots; mind design space is big!

I have not yet watched a movie where humans are casually obliterated by a superior force, be that a GAI or a technologically superior alien species. At least some of the humans always seem to have a fighting chance. The odds are overwhelming of course, but the enemy always has a blind spot that can be exploited. You list some of them here. They are just the kind of thing McKay deploys successfully against advanced nanotechnology. Different shows naturally give the AI different exploitable weaknesses. For the sake of the story such AIs are almost always completely blind to the most of the obvious weaknesses of humanity.

The whole 'overcome a superior enemy by playing to your strengths and exploiting their weakness' makes for great viewing but outside of the movies it is far less likely to play a part. The chance of creating an uFAI that is powerful enough to be a threat and launch some kind of attack and yet not be able to wipe out humans is negligible outside of fiction. Chimpanzees do not prevail over a civilisation with nuclear weapons. And no, the fact that they can beat us in unarmed close combat does not matter. They just die.

Comment author: Johnicholas 21 November 2009 05:11:15PM 0 points [-]

Yes, this is movie-plot-ish-thinking in the sense that I'm proposing that superintelligences can be both dangerous and defeatable/controllable/mitigatable. I'm as prone to falling into the standard human fallacies as the next person.

However, the notion that "avoid strength, attack weakness" is primarily a movie-plot-ism seems dubious to me.

Here is a more concrete prophesy, that I hope will help us communicate better:

Humans will perform software experiments trying to harness badly-understood technologies (ecosystems of self-modifying software agents, say). There will be some (epsilon) danger of paperclipping in this process. Humans will take precautions (lots of people have ideas for precautions that we could take). It is rational for them to take precautions, AND the precautions do not completely eliminate the chance of paperclipping, AND it is rational for them to forge ahead with the experiments despite the danger. During these experiments, people will gradually learn how the badly-understood technologies work, and transform them into much safer (and often much more effective) technologies.

Comment author: wedrifid 21 November 2009 05:45:39PM 1 point [-]

However, the notion that "avoid strength, attack weakness" is primarily a movie-plot-ism seems dubious to me.

That certainly would be dubious. Avoid strength, attack weakness is right behind 'be a whole heap stronger' as far as obvious universal strategies go.

Humans will perform software experiments trying to harness badly-understood technologies (ecosystems of self-modifying software agents, say). There will be some (epsilon) danger of paperclipping in this process. Humans will take precautions (lots of people have ideas for precautions that we could take). It is rational for them to take precautions, AND the precautions do not completely eliminate the chance of paperclipping, AND it is rational for them to forge ahead with the experiments despite the danger. During these experiments, people will gradually learn how the badly-understood technologies work, and transform them into much safer (and often much more effective) technologies.

If there are ways to make it possible to experiment and make small mistakes and minimise the risk of catastrophe then I am all in favour of using them. Working out which experiments are good ones to do so that people can learn from them and which ones will make everything dead is a non-trivial task that I'm quite glad to leave to someone else. Given that I suspect both caution and courage to lead to an unfortunately high probability of extinction I don't envy them the responsibility.

AND it is rational for them to forge ahead with the experiments despite the danger.

Possibly. You can't make that conclusion without knowing the epsilon in question and the alternatives to such experimentation. But there are times when it is rational to go ahead despite the danger.

Comment author: timtyler 21 November 2009 06:02:52PM *  1 point [-]

The fate of most species is extinction. As the first intelligent agents, people can't seriously expect our species to last for very long. Now that we have unleashed user-modifiable genetic materials on the planet, DNA's days are surely numbered. Surely that's a good thing. Today's primitive and backwards biotechnology is a useless tangle of unmaintainable spaghetti code that leaves a trail of slime wherever it goes - who would want to preserve that?

Comment author: CronoDAS 21 November 2009 06:14:53AM 0 points [-]

I have not yet watched a movie where humans are casually obliterated by a superior force, be that a GAI or a technologically superior alien species.

You didn't see the Hitchhiker's Guide to the Galaxy film? ;)

Comment author: wedrifid 21 November 2009 06:22:23AM 0 points [-]

:) Well, maybe substitute 'some' for 'one' in the next sentence.

Comment author: timtyler 21 November 2009 07:19:32AM 1 point [-]

http://en.wikipedia.org/wiki/Invasion_of_the_Body_Snatchers_(1978_film)

...apparently has everyone getting it fairly quickly at the hands of aliens.

Comment author: loqi 21 November 2009 02:23:45AM 3 points [-]

I'm pretty good at beating my computer at chess, even though I'm an awful player. I challenge it, and it runs out of time - apparently it can't tell that it's in a competition, or can't press the button on the clock.

You know this trick too? You wouldn't believe how many quadriplegics I've beaten at chess this way.

Comment author: timtyler 21 November 2009 07:22:32AM 1 point [-]

A superintelligence can reasonably be expected to proactively track down its "blind spots" and eradicate them - unless it's "blind spots" are very carefully engineered.

Comment author: Johnicholas 21 November 2009 05:42:33PM 1 point [-]

As I understand your argument, you start with an artificial mind, a potential paperclipping danger, and then (for some reason? why does it do this? Remember, it doesn't have evolved motives) it goes through a blind-spot-eradication program. Afterward, all the blind spots remaining would be self-shadowing blind spots. This far, I agree with you.

The question of how many remaining blind spots, or how big they are has something to do with the space of possible minds and the dynamics of self-modification. I don't think we know enough about this space/dynamics to conclude that remaining blind spots would have to be carefully engineered.

Comment author: wedrifid 21 November 2009 05:54:37PM 4 points [-]

(for some reason? why does it do this? Remember, it doesn't have evolved motives) it goes through a blind-spot-eradication program.

You have granted a GAI paperclip maximiser. It wants to make paperclips. That's all the motive it needs. Areas of competitive weakness are things that may make it get destroyed by humans. If it is destroyed by humans less paperclips will be made. It will eliminate its weaknesses with high priority. It will quite possibly eliminate all the plausible vulnerabilities and also the entire human species before it makes a single paperclip. That's just good paperclip maximising sense.

Comment author: Johnicholas 21 November 2009 07:20:00PM 4 points [-]

As I understand your thought process (and Steve Omohundro's), you start by saying "it wants to make paperclips", and then, in order to predict its actions, you recursively ask yourself "what would I do in order to make paperclips?".

However, this recursion will inject a huge dose of human-mind-ish-ness. It is not at all clear to me that "has goals" or "has desires" is a common or natural feature of mind space. When we study powerful optimization processes - notably, evolution, but also annealing and very large human organizations - we generally can model some aspects of their behavior as goals or desires, but always with huge caveats. The overall impression that we get of these processes, considered as minds, is that they're insane.

Insane is not the same as stupid, and it's not the same as safe.

Comment author: timtyler 21 November 2009 05:53:41PM *  0 points [-]

This is because of the natural drives that we can reasonably expect many intelligent agents to exhibit - see:

http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/

http://selfawaresystems.com/2009/02/18/agi-08-talk-the-basic-ai-drives/

Comment author: Vladimir_Nesov 18 November 2009 08:28:56PM *  1 point [-]

Unfriendly AGI, despite its potential power, may well have huge blind spots; mind design space is big!

You may be right, but I don't think it's a very fruitful idea: what exactly do you propose doing? Also, building of a FAI is a distinct effort from e.g. healing malaria or fighting specific killer robots (with the latter being quite hypothetical, while at least the question of technically understanding FAI seems inevitable).

This may be possible if an AGI has a combination of two features: it has significant real-world capabilities that make it dangerous, yet it's insane or incapable enough to not be able to do AGI design. I don't think it's very plausible, since (1) even Nature was able to build us, given enough resources, and it has no mind at all, so it shouldn't be fundamentally difficult to build an AGI (even for an irrational proto-AGI) and (2) we are at the lower threshold of being dangerous to ourselves, yet it seems we are at the brink of building an AGI already. Having an AGI dangerous (extinction risk dangerous), and dangerous exactly because of its intelligence, yet not AGI-building-capable doesn't seem to me unlikely. But maybe possible for some time.

Now, consider the argument about humans being at the lowest possible cognitive capability to do much of anything, applied to proto-AGI-designed AGIs. AGI-designed AGIs are unlikely to be exactly as dangerous as the designer AGI, they are more likely to be significantly more or less dangerous, with "less dangerous" not being an interesting category, if both kinds of designs occur over time. This expected danger adds to the danger of the original AGI, however inapt they themselves may be. And at some point, you get to an FAI-theory-capable-AGI that builds something rational, not once failing all the way to the end of times.

Comment author: Johnicholas 19 November 2009 12:52:26AM *  1 point [-]

I'd like to continue this conversation, but we're both going to have to be more verbose. Both of us are speaking in very compressed allusive (that is, allusion-heavy) style, and the potential for miscommunication is high.

"I don't think it's a very fruitful idea: what exactly do you propose doing?" My notion is that SIAI in general and EY in particular, typically work with a specific "default future" - a world where, due to Moore's law and the advance of technology generally, the difficulty of building a "general-purpose" intelligent computer program drops lower and lower, until one is accidentally or misguidedly created, and the world is destroyed in a span of weeks. I understand that the default future here is intended to be a conservative worst-case possibility, and not a most-probable scenario.

However, this scenario ignores the number and power of entities (such as corporations, human-computer teams, and special-purpose computer programs) and which are more intelligent in specific domains than humans. It ignores their danger - human potential flourishing can be harmed by other things than pure software - and it ignores their potential as tools against unFriendly superintelligence.

Correcting that default future to something more realistic seems fruitful enough to me.

"Technically understanding FAI seems inevitable." What? I don't understand this claim at all. Friendly artificial intelligence, as a theory, need not necessarily be developed before the world is destroyed or significantly harmed.

"This may be possible" What is the referent of "this"? Techniques for combating, constraining, controlling, or manipulating unFriendly superintelligence? We already have these techniques. We harness all kinds of things which are not inherently Friendly and turn them to our purposes (rivers, nations, bacterial colonies). Techniques of building Friendly entities will grow directly out of our existing techniques of taming and harnessing the world, including but not limited to our techniques of proving computer programs correct.

I am not sure I understand your argument in detail, but from what I can tell, your argument is focused "internal" to the aforementioned default future. My focus is on the fact that many very smart AI researchers are dubious about this default future, and on trying to update and incorporate that information.

Comment author: Jordan 19 November 2009 01:38:21AM 0 points [-]

However, this scenario ignores the number and power of entities (such as corporations, human-computer teams, and special-purpose computer programs) and which are more intelligent in specific domains than humans.

Good point. Even an Einstein level AI with 100 times the computing power of an average human brain probably wouldn't be able to be beat Deep Blue at chess (at least not easily).

Comment author: RobinZ 18 November 2009 05:19:04PM 0 points [-]

A helpful tip: early computer chess programs were very bad at "doing nothing, and doing it well". I believe they are bad at Go for similar reasons.

Comment author: Eliezer_Yudkowsky 18 November 2009 03:24:28AM 5 points [-]

By "deal with" I'm including prevailing over superintelligences, without throwing up hands and saying "it's smarter than me".

throws up hands

Not every challenge is winnable, you know.

Comment author: JamesAndrix 18 November 2009 10:11:43AM 0 points [-]

Impossible?

Are you saying a human can't beat a group mind or are you and Johnicholas using different meanings of superintelligence?

Also, what if we're in a FAI without a nonperson predicate?

Comment author: timtyler 18 November 2009 09:32:08AM *  -2 points [-]

Companies and governments are probably the most likely organisations to play host superintelligent machines in the future.

They are around today - and I think many of the efforts on machine morality would be better spent on making those organisations more human-friendly. "Don't be evil" is just not good enough!

I think reputation systems would help a lot. See my "Universal karma" video: http://www.youtube.com/watch?v=tfArOZVKCCw

Comment author: DanArmak 18 November 2009 05:32:50PM 1 point [-]

"Don't be evil" is just not good enough!

Because all except (so it is claimed) one company and all governments bar none do not even pretend to embrace this attitude.

I think reputation systems would help a lot.

OK, so my government gets low karma. So what? How does that stop them for doing whatever they want to for years to come?

If you suggest it would cause members of parliament to vote no confidence and cause early elections - something which only applies in a many-parties democracy - then I suggest that in such a situation no government could remain stable for long. There'd be a new cause celebre every other week, and a government only has to lose one vote to fall.

And if the public, through karma, could force government to act in certain ways without going through elections, then we'd have direct democracy with absolute-majority-rule. A system that's even worse than what we have today.

Comment author: timtyler 18 November 2009 06:07:49PM 0 points [-]

Of course, governments and companies have reputations today.

There are few enough countries that people can keep track of their reputations reasonably easily when it comes to trade, travel and changing citizenship.

It is probably companies where reputations are needed the most. You can search - and there's resources like:

http://www.dmoz.org/Society/Issues/Business/Allegedly_Unethical_Firms/

...but society needs more.

Comment author: whpearson 17 November 2009 05:23:03PM *  0 points [-]

I'm really looking for a justification of the nuclear reactor metaphor for intelligence amplifying intelligence, on the software level.

AI might explode sure, but exponential intelligence amplification on the software level pretty much guarantees it on the first AI rather than us having to wait around and possibly merge before the explosion.

Comment author: timtyler 17 November 2009 10:56:14PM -2 points [-]

Intelligence is building on itself today. That's why we see the progress we do. See:

http://en.wikipedia.org/wiki/Intelligence_augmentation

If you want to see an hardware explosion, look to Moore's law. For a software explosion, the number of lines of code being written is reputedly doubling even faster than every 18 months.

Comment author: whpearson 17 November 2009 11:26:53PM 1 point [-]

I want to see an explosion in the efficacy of software, not simply the amount that is written.

Comment author: timtyler 18 November 2009 12:03:45AM *  1 point [-]

Software is gradually getting better. If you want to see how fast machine intelligence software is progressing, one reasonably-well measured area is chess/go ratings.

Comment author: DanArmak 18 November 2009 05:19:23PM 1 point [-]

How can progress in such a narrow problem be representative of the efficacy of software either in some general sense or versus other narrow problems?

Also: what is the improvement over time of machine chess playing ability due to software changes once you subtract hardware improvements? I remember seeing vague claims that chess performance over the decades stayed fairly true to Moore's Law, i.e. scaled with hardware. As a lower bound this is entirely unsurprising, since naive chess implementations (walk game tree to depth X) scale easily with both core speed and number of cores.

Comment author: Johnicholas 17 November 2009 06:13:51PM 0 points [-]

As I understand it, the nuclear reactor metaphor is simply another way of saying "explosive" or "trending at least exponentially".

Note that most (admittedly fictional) descriptions of intelligence explosion include "bootstrapping" improved hardware (e.g. Greg Egan's Crystal Nights)

Comment author: Thomas 17 November 2009 05:34:19PM -1 points [-]

The computation is not a "pure process", it has its physical side and this may be used as a matter transformer by some software. There from a FOOM might appear.

I asked that Eliezer Yudkowsky in the questions for him, but nearly nobody noticed by vote it up.

Comment author: wedrifid 17 November 2009 07:19:48PM *  0 points [-]

Hang on, I voted this up because it was a good point but on second glance it isn't a point that is at all relevant to what whpearson is asking.

Comment author: Zack_M_Davis 17 November 2009 04:33:55PM 3 points [-]

Robin criticizes Eliezer for not having written up his arguments about the Singularity in a standard style and submitted them for publication. Others, too, make the same complaint: the arguments involved are covered over such a huge mountain of posts that it's impossible for most outsiders to seriously evaluate them.

Did everyone forget about "Artificial Intelligence as a Positive and Negative Factor in Global Risk"?

Comment author: Kaj_Sotala 17 November 2009 05:09:16PM 8 points [-]

That one's useful, but IMO, as an introduction document it needs a lot of work. For instance, the main thesis should be stated a lot earlier and clearer. As it is, it spends several pages making different points about the nature of AI before saying anything about why AI is worth discussing as a risk in the first place. That comes as late as page 17, and much of that is an analogue to the nuclear bomb. It spends a lot of time arguing against the proposition that a hard takeoff is impossible, but not much time arguing for the proposition that a hard takeoff is likely, which is a major failing if it's supposed to convince people.

Mostly the paper suffers from the problem of being too spread out: it doesn't really give strong support to a narrow set of core claims, instead giving weak support to a wide set of core and non-core claims. I have a memory of reading the thing back when I didn't know so much about the Singularity, and thinking it was good, even though it took a long time to get to the point and had stuff that seemed unnecessary. It was only later on when I re-read the paper that I realized how those "unnecessary" parts connected with other pieces of FAI theory - but of course, by that time I wasn't really an outsider anymore.

Comment author: timtyler 17 November 2009 11:24:41PM *  2 points [-]
Comment author: timtyler 17 November 2009 07:44:14PM *  1 point [-]

Aubrey argues for "the singularity" here:

"The singularity and the Methuselarity: similarities and differences" - by Aubrey de Grey

http://www.sens.org/files/sens/FHTI07-deGrey.pdf

He uses the argument from personal incredulity though - one of the weakest forms of argument known.

He says:

"But wait – who’s to say that progress will remain “only” exponential? Might not progress exceed this rate, following an inverse polynomial curve (like gravity) or even an inverse exponential curve? I, for one, don’t see why it shouldn’t. If we consider specifically the means whereby the Singularity is most widely expected to occur, namely the development of computers with the capacity for recursive improvement of their own workings, I can see no argument why the rate at which such a computer would improve itself should not follow an inverse exponential curve, i.e. one in which the time taken to achieve a given degree of improvement takes time X, the time taken to repeat that degree of improvement is X/2, then X/4 and so on."

My reply: to build a fast computer you don't just need to perform computations quickly. You also need to be able to design and perform real-world experiments and tests. While sensor and motor capabilities are improving, they are not doing so with the same doubling time as exists for computers. Consequently, progress in these fields (and overall progress) is correspondingly slower.

Then there's the influence of limits. Clock speed has already run into diminishing returns. By the time the slower-doubling systems have time to double very many times, the faster-doubling ones will have maxed-out - and will be hindering overall progress.

IMO, nobody seems to have thought the "singularity" idea through :-(

Comment author: Tyrrell_McAllister 17 November 2009 09:31:46PM 1 point [-]

I haven't read the whole essay, but the portion that you quoted isn't an argument from incredulity.

An argument from incredulity has the form "Since I can't think of an argument for assigning P low probability, I should assign P high probability.".

Aubrey's argument has the form "Since I can't think of an argument for assigning P low probability, I shouldn't assign P low probability.".

Comment author: timtyler 17 November 2009 09:51:13PM *  1 point [-]

He's expressing incredulity - and then arguing from that. He goes on to assume that this "singularity" thing happens for much of the rest of the paper.

Comment author: StefanPernar 18 November 2009 04:02:18AM *  1 point [-]

Very constructive proposal Kaj. But...

Since it appears (do correct me if I'm wrong!) that Eliezer doesn't currently consider it worth the time and effort to do this, why not enlist the LW community in summarizing his arguments the best we can and submit them somewhere once we're done?

If Eliezer does not find it a worthwhile investment of his time - why should we?

Comment author: Eliezer_Yudkowsky 18 November 2009 04:29:07AM 7 points [-]

In general: Because my time can be used to do other things which your time cannot be used to do; we are not fungible.

(As of this comment being typed, I'm working on a rationality book. This is not something that anyone else can do for me.)

Comment author: StefanPernar 18 November 2009 05:08:09AM 0 points [-]

In general: Because my time can be used to do other things which your time cannot be used to do; we are not fungible.

This statement is based on three assumptions: 1) What you are doing instead is in fact more worthy of your attention than your contribution here 2) I could not do what you are doing as least as well as you 3) I do not have other things to do that are at least as worthy of my time

None of those three I am personally willing to grant at this point. But surely that is not the case for all the others around here.

Comment author: kurige 18 November 2009 06:19:09AM *  2 points [-]

1) You can summarize arguments voiced by EY.
2) You cannot write a book that will be published under EY's name.
3) Writing a book takes a great deal of time and effort.

You're reading into connotation a bit too much.

Comment author: StefanPernar 18 November 2009 06:45:02AM 0 points [-]

2) You cannot write a book that will be published under EY's name.

Its called ghost writing :-) but then again the true value add lies in the work and not in the identity of the author. (discarding marketing value in the case of celebrities)

Your reading into connotation a bit too much.

I do not think so - am just being German :-) about it: very precise and thorough.

Comment author: Kaj_Sotala 18 November 2009 05:42:52PM 0 points [-]

I think his arguments are worthwhile and important enough to be heard by an audience that is as wide as possible, regardless of whether or not he feels like writing up the arguments in an easily digestible form.

Comment author: alyssavance 18 November 2009 01:47:46AM 1 point [-]

Hmmm... Maybe you could base some of it off of Eliezer's Future Salon talk (http://singinst.org/upload/futuresalon.pdf)? That's only about 11K words (sans references), while his book chapters are ~40K words and his OB/LW posts are hundreds of thousands of words.

Comment author: soreff 08 December 2009 04:35:44PM *  1 point [-]

Any thoughts on what the impact of the http://www.research.ibm.com/deepqa/ IBM Watson Deepqa project would be on a Foom timescale, if it is successful (in the sense of approximate parity with human competitors)? My impression was that classical AI failed primarily because of brittle closed-world approximations, and this project looks like it (if successful) would largely overcome those obstacles. For instance, it seems like one could integrate a deepqa engine with planning and optimization engines in a fairly straightforward way. To put it another way, in the form of an idea futures proposition: conditional on deepqa being fully successful at a human-equivalent level, what are the odds of a human equivalent UFAI in all areas needed for recursive self-improvement plus replication without human intervention in say 5 years after that point? Best wishes, -Jeff

Comment author: JoshuaFox 18 November 2009 11:41:50AM *  0 points [-]

Kaj, great idea

in a standard style and submitted them for publication

This will be one of the greater challenges; we know the argument and how to write well, but each academic discipline has rigid rules for style in publications. This particular journal, with its wide scope, may be a bit more tolerant, but in general learning the style is important if one wants to influence academia.

I imagine that one will have to go beyond the crowdsourcing approach in achieving this.

If you are coordinating, let me know if and how I can help.

Comment author: Kaj_Sotala 18 November 2009 05:41:31PM 1 point [-]

I was mostly thinking that people would jointly write a draft article, which the more experienced writers would then edit to conform with a more academic style, if necessary.

Comment author: StefanPernar 18 November 2009 04:14:19AM 0 points [-]

More recent criticism comes from Mike Treder - managing director of the Institute for Ethics and Emerging Technologies in his article "Fearing the Wrong Monsters" => http://ieet.org/index.php/IEET/more/treder20091031/

Comment author: righteousreason 21 November 2009 08:40:54PM *  -2 points [-]

==Re comments on "Singularity Paper"== Re comments, I had been given to understand that the point of the page was to summarize and cite Eliezer's arguments for the audience of ''Minds and Machines''. Do you think this was just a bad idea from the start? (That's a serious question; it might very well be.) Or do you think the endeavor is a good one, but the writing on the page is just lame? --User:Zack M. Davis 20:19, 21 November 2009 (UTC)

(this is about my opinion on the writing in the wiki page)

No, just use his writing as much as possible- directly in the text of the paper. Whole articles/posts in sequence for the whole paper would be best, or try to copy-paste together some of the key points of a series of articles/posts (but do you really want to do that and leave out the rich, coherent, consistent explanation that these points are surrounded in?)

My comments may seem to imply that we would essentially be putting together a book. That would be an AWESOME book... we could call it "Intelligence Explosion".

If someone ended up doing a book like that, they might as well include a section on FAI. If SIAI produces a relevant FAI paper, that could be included (or merged) into the FAI section

SEE THIS:

http://www.imminst.org/forum/index.php?showtopic=35318&hl=