Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Hanson Debating Yudkowsky, Jun 2011

14 Post author: XiXiDu 03 July 2011 04:59PM

On Wednesday I debated my ex-co-blogger Eliezer Yudkowsky at a private Jane Street Capital event (crude audio here, from 4:45; better video here [as of July 14]).

I “won” in the sense of gaining more audience votes — the vote was 45-40 (him to me) before, and 32-33 after the debate. That makes me two for two, after my similar “win” over Bryan Caplan (42-10 before, 25-20 after). This probably says little about me, however, since contrarians usually “win” such debates.

Our topic was: Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world. He was pro, I was con. We also debated this subject here on Overcoming Bias from June to December 2008. Let me now try to summarize my current position.

[...]

It thus seems quite unlikely that one AI team could find an architectural innovation powerful enough to let it go from tiny to taking over the world within a few weeks.

Link: overcomingbias.com/2011/07/debating-yudkowsky.html

Comments (78)

Comment author: John_Maxwell_IV 04 July 2011 04:18:54AM *  8 points [-]

Can someone who agrees with Yudkowsky do an extended summary of Yudkowsky's position and arguments in text, the way Hanson summarized his position and arguments in text?

Comment author: XiXiDu 04 July 2011 12:12:35PM 8 points [-]
Comment author: Sewing-Machine 03 July 2011 11:28:35PM 18 points [-]

Hanson often takes his turn to speak this way: he summarizes Yudkowsky's last argument, in a way that at least superficially does not seem unfair or tendentious, then explains why he doesn't find it compelling, then explains why his own position is more compelling.

Yudkowsky seems to respond to Hanson's points without summarizing them first.

I find Hanson to be hugely more effective in the recording. Is it because of this? I was less sympathetic to Yudkowsky's point of view before I started listening, so it's hard for me to tell if this is an illusion.

Comment author: Douglas_Knight 05 July 2011 03:32:51AM 5 points [-]

In this recording (and to a lesser extent in the debate with Caplan), Hanson is much louder than his opponent, probably because of microphone placement. I expect that this makes him more effective, though I haven't listened to it much.

Comment author: wedrifid 04 July 2011 06:45:35AM 3 points [-]

I find Hanson to be hugely more effective in the recording. Is it because of this?

Partly. Hanson is a far more powerful rhetorician then Eliezer. He speaks with the kind of language that is far more persuasive to the majority of people. It is only a narrow audience with particularly high standards with respect to judging "tendentious" tendencies for whom his style backfires.

Comment author: XiXiDu 04 July 2011 12:19:18PM *  4 points [-]

Hanson is a far more powerful rhetorician then Eliezer. He speaks with the kind of language that is far more persuasive to the majority of people.

Off-topic: How do you like the style of Nick Bostrom? Here is another video of him.

Comment author: timtyler 08 September 2011 07:21:26AM 5 points [-]
Comment author: timtyler 06 July 2011 09:38:22PM *  4 points [-]

72:00 - Hanson cites the AAAI white paper which rails against alarmism.

That seems more likely to be a tussle over funding. If the panic-stricken screechers make enough noise, it might adversely affect their member's funding. Something rather like that has happened before:

It was rumoured in some of the UK national press of the time that Margaret Thatcher watched Professor Fredkin being interviewed on a late night TV science programme. Fredkin explained that superintelligent machines were destined to surpass the human race in intelligence quite soon, and that if we were lucky they might find human beings interesting enough to keep us around as pets. The rumour is that Margaret Thatcher decided that the ``artificial intelligentsia'' whom she was just proposing to give lots of research funds under the Alvey Initiative were seriously deranged.

Comment author: timtyler 06 July 2011 04:20:00PM 3 points [-]

Legg's Is there an Elegant Universal Theory of Prediction? is somewhat relevant to parts of this discussion.

Comment author: Vladimir_Nesov 03 July 2011 10:32:45PM *  3 points [-]

Hanson seems to agree that if we get human-level agents that are cheap to run, this gets us a local takeover. I don't think that having cheap chimp-level agents widely available at that time overturns the advantage of gaining access to cheap human-level agents. So if we grant that the capability of AIs gets increased gradually and publicly, all that a local group needs to take over the world is make the step from chimp-level state-of-the-art agents to human-level agents before any other group does that. If chimp-level agents are not that different from human-level agents, this could be a relatively simple step, that a single group is capable of making in secret.

(I understand the limitations of this classification, but we could as well see this as an analogy to how the evolution developed humans from their ancestors with just a little bit of extra capacity, and that little bit mattered a lot, given sufficient time to express its capability. The local team is not developing the human culture, it is making humans where we previously had only chimps. And then it gives the humans cheap computing power to run on and develop their powerful culture.)

Comment author: Sewing-Machine 04 July 2011 12:27:01AM 8 points [-]

I don't have an intuition for what would happen if you ran a chimp-level intelligence very fast. The ratio Yudkowsky mentioned in the recording was 2500 years of human-in-skull thinking = 8 hours of human-in-laptop thinking. Is it completely obvious that 2500 years of chimp thinking would yield nothing interesting or dangerous?

Chimps haven't accomplished much in the last 2500 years but that's at least partly because they don't pass on insights between generations. Can we stipulate 2500 years of chimp memory, too?

Comment author: Vladimir_Nesov 04 July 2011 02:25:57AM 11 points [-]

Modifiers like "new" or "old" in titles or filenames quickly become unhelpful. It's generally better to use dates (in this case, something like "Jun 2011" would do the trick).

Comment author: Normal_Anomaly 03 July 2011 06:10:11PM *  6 points [-]

In both of these debates, the change in "margin of victory" between people was smaller than the number of people who voted the first time but not the second. In the debate with Yudkowsky, Hanson's margin of victory went from -5 to 1, and 20 voters dropped out. In the debate with Caplan the margin went from 32 to 5--actually getting smaller--and 17 voters dropped out. I'm not sure if we can even determine a "winner" in terms of audience popularity, with that many disappearing votes. Is it normal in debates for large numbers of audience members to not vote at the end?

Comment author: Douglas_Knight 03 July 2011 09:32:11PM *  0 points [-]

vs Caplan, the margin went from -32 to -5, with 7 people dropping out. Do you have some other source of numbers?

If people leave early, that is an extra source of noise, but probably unbiased. If a non-voter is expressing uncertainty, that is a useful source of information, although it would be better to know how many abstain and better if they explicitly expressed uncertainty. I would expect people to be more bashful before the debate than after and am surprised that the number of votes went down.

ETA: the recording of the Eliezer-Hanson debate says that this was an increase in number of undecided people.

Comment author: Normal_Anomaly 03 July 2011 09:36:27PM 1 point [-]

Oops, I typed 12 when I meant 32. But I think it's positive 32 and 5, not negative, because he says he won.

Comment author: Douglas_Knight 04 July 2011 02:51:19AM 1 point [-]

Try reading it again for the signs. Moreover, when he says that contrarian positions win, he means that they shift people towards them, not that they win outright. This is probably because most people have never heard actual arguments for the contrarian position.

Comment author: wallowinmaya 04 July 2011 02:36:53PM 4 points [-]

My two cents:

  1. Is it really useful to debate such a difficult and completely unintuitive concept in 90 minutes? It took me at least 20 hours pure reading time to finish the AI-foom debate, which consists of over 50 blog-posts.
  2. Hanson uses simpler analogies than Yudkowsky and tries to explain basic concepts, whereas Yudkowsky's information-dense arguments often aim too high, especially considering the short time.
Comment author: Vaniver 05 July 2011 03:30:07AM 3 points [-]

If anybody can understand this after 90 minutes, it's the guys at Jane Street.

Comment author: JenniferRM 06 July 2011 06:56:09AM *  4 points [-]

The primary impact appears to have been to increase the undecided votes which (assuming a high quality audience) implies to me that many members of the audience were somewhat irrationally polarized to begin with (as might be reasonable) and many people heard things from "the other side" during the debate that they had never heard before and updated towards 50/50 ignorance based on the debate. (Perhaps there is some wishful thinking in this model? Hard to tell.)

If this model of the debate and audience is right, it suggests that no one (neither Hanson nor Eli nor anyone else I know of), has a clearly correct model for singularity issues with slots for evidence that has been tracked down and dropped into place to give a clean and clear answer. Basically, it sound to me like most very smart people who've thought about the singularity are still confused, and more of them are beginning to notice their confusion.

Comment author: MatthewBaker 05 July 2011 11:20:57PM 0 points [-]

Some people read really fast :)

Comment author: aletheilia 06 July 2011 08:13:46AM 2 points [-]

How about a LW poll regarding this issue?

(Is there some new way to make one, since the site redesign, or are we still at vote-up-down-karma-balance pattern?)

Comment author: wedrifid 03 July 2011 05:54:30PM 0 points [-]

Hanson's debating success is all the more impressive given that he was fighting with a handicap. Imagine how potent his debating would be if he was actually arguing for a correct position!

Comment author: AlexMennen 04 July 2011 01:57:04AM 3 points [-]

When the judges are human, defending an incorrect position is not much of a handicap in debates on issues to which human intuition is poorly attuned.

Comment author: Manfred 05 July 2011 08:54:36PM 2 points [-]

lol. Anti-downvoted.

Comment author: XiXiDu 03 July 2011 06:52:53PM *  2 points [-]

Imagine how potent his debating would be if he was actually arguing for a correct position!

He will probably start arguing for the correct position once it is supported enough as not to be destroyed by an incorrect position.

Comment author: wedrifid 03 July 2011 07:16:02PM 3 points [-]

He will probably start arguing for the correct position once it is supported enough as not to be destroyed by incorrect positions.

The main thing I can recall from from the 2008 debate mentioned was Hanson's position being essentially destroyed by Hanson via support that made no sense, making Eliezer largely redundant.

Comment author: XiXiDu 03 July 2011 07:47:15PM 2 points [-]

Hanson's position being essentially destroyed by Hanson via support that made no sense...

As far as I can tell Hanson does not disagree with Yudkowsky except for the probability of risks from AI. Yudkowsky says that existential risks from AI are not under 5%. Has Yudkowsky been able to support this assertion sufficiently? Hanson only needs to show that it is unreasonable to assume that the probability is larger than 5% and my personal perception is that he was able to do so.

I have already posted various arguments for why I believe that the case for risks from AI, and especially recursive self-improvement (explosive), is not as strongly supported as some people seem to think. I haven't come across a good refutation, or a strong argument to the contrary.

You are of course free to suspect and declare that I was already given sufficient evidence and that my motives are not allowing me to admit that I am wrong. I won't perceive anything like that as a personal attack and ask everyone not to downvote you for any such statements. Think I am a troll or an idiot, let me know, I want to know :-)

Comment author: hairyfigment 04 July 2011 02:57:55AM *  2 points [-]

A while back I posted this minimalist account of Eliezer's case for the importance of FAI to human survival. (Claim B technically seems too specific if you want to talk about the existential risk as a whole, but I think it reflects his view.)

So far I can't tell if you agree that each claim has easily more than .5 probability given the evidence, nor if you share my view that Claim A as separate from the rest has P close to 1. In particular, you said here that you believe:

if you waste too much time with spatiotemporal bounded versions then someone who is ignorant of friendliness will launch one that isn't constrained that way.

By the same principle, the speed or slowness of FOOM doesn't matter in the long run unless some force with the power to stop it does so, and unless this happens every single time someone creates an unFriendly AI with the power to self-modify. I have almost no confidence that humanity in general will learn from past mistakes (and precious little confidence in the subset that could write the second or third AGI). So I think we need to look at the cumulative chance for Claim B, Claim C, and perhaps even D.

Even so, it seems possible that the actual risk stays within 5%. Maybe you think some form of FAI, such as Friendly uploads, will prove easy once we get the capacity for some form of AGI. Maybe you think we seem likely to kill ourselves with X before then. Maybe you think some other force(s) will stop each and every AGI. If so, I'd like to hear your reasoning.

And if not, if you want to argue against my claims in some other way, please do so without identifying them with a more specific storyline.

ETA: I apparently forgot how to use links. I believe this means I should go eat or sleep. Take that as you will.

Comment author: XiXiDu 04 July 2011 11:56:53AM *  1 point [-]

In particular, you said here that you believe...

Yes, I said that I believe that even sub-human level AI pose an existential risk. At the same time I am highly skeptical of FOOM. So why don't I agree with Eliezer outright anyway? Because the risks from AI that I perceive to be a possibility are not something you can solve by inventing provable "friendliness". How are you going to make a sophisticated monitoring system friendly? Why would people want to make it friendly? How are you going to make a virus with sub-human level general intelligence friendly? Why would one do that? Risks from AI are a broad category that need meta-solutions that involve preemptive political and security measures. You need to make sure that the first intelligent surveillance systems are employed transparently and democratically so that everyone can monitor the world for the various risks ahead. We need a global immune system that keeps care that nowhere someone gets ahead of everyone else.

Comment author: Rain 05 July 2011 06:50:04PM *  2 points [-]

Have you taken your own survey and published the results somewhere? Or is it only for AI researchers? It seems like there are a great deal of hidden assumptions on all sides which make these discussions go off the tracks very quickly. Some kind of basic survey with standard probability estimates might easily show where views differ.

So why don't I agree with Eliezer outright anyway? Because the risks from AI that I perceive to be a possibility are not something you can solve by inventing provable "friendliness".

I agree that friendliness is a long shot. If you know of a better solution, please let me know.

How are you going to make a sophisticated monitoring system friendly?

By developing a theory of friendliness and implementing it in software.

Why would people want to make it friendly?

Because unfriendly things are bad.

How are you going to make a virus with sub-human level general intelligence friendly?

By developing a theory of friendliness and implementing it in software.

Why would one do that?

Because unfriendly things are bad.

Risks from AI are a broad category that need meta-solutions that involve preemptive political and security measures. You need to make sure that the first intelligent surveillance systems are employed transparently and democratically so that everyone can monitor the world for the various risks ahead. We need a global immune system that keeps care that nowhere someone gets ahead of everyone else.

Sounds like a job for CEV and a friendly AI to run it on.

Comment author: XiXiDu 06 July 2011 11:00:23AM 1 point [-]

Have you taken your own survey and published the results somewhere?

Yes I have done so. But I don't trust my ability to make correct probability estimates, don't trust the overall arguments and methods and don't know how to integrate that uncertainty into my estimates. It is all too vague.

There sure are a lot of convincing arguments in favor of risks from AI. But do arguments suffice? Nobody is an expert when it comes to intelligence. Even worse, I don't think anybody knows much about artificial general intelligence.

My problem is that I fear that some convincing blog posts are simply not enough. Just imagine all there was to climate change was someone with a blog who never studied the climate but instead wrote some essays about how it might be physical possible for humans to cause a global warming. Not enough, the same person then goes on to make further inferences based on the implications of those speculations. Am I going to tell everyone to stop emitting CO2 because of that? Hardly! Or imagine that all there was to the possibility of asteroid strikes was someone who argued that there might be big chunks of rocks out there which might fall down on our heads and kill us all, inductively based on the fact that the Earth and the moon are also a big rocks. Would I be willing to launch a billion dollar asteroid deflection program solely based on such speculations? I don't think so. Luckily, in both cases, we got a lot more than some convincing arguments in support of those risks.

Another example: If there were no studies about the safety of high energy physics experiments then I might assign a 20% chance of a powerful particle accelerator destroying the universe based on some convincing arguments put forth on a blog by someone who never studied high energy physics. We know that such an estimate would be wrong by many orders of magnitude. Yet the reason for being wrong would largely be a result of my inability to make correct probability estimates, the result of vagueness or a failure of the methods I employed. The reason for being wrong by many orders of magnitude would have nothing to do with the arguments in favor of the risks, as they might very well be sound given my epistemic sate and the prevalent uncertainty.

In summary: I believe that mere arguments in favor of one risk do not suffice to neglect other risks that are supported by other kinds of evidence. I believe that logical implications of sound arguments should not reach out indefinitely and thereby outweigh other risks whose implications are fortified by empirical evidence. Sound arguments, predictions, speculations and their logical implications are enough to demand further attention and research, but not much more.

I agree that friendliness is a long shot. If you know of a better solution, please let me know.

If there was a risk that might kill us with a probability of .7 and another risk with .1 while our chance to solve the first one was .0001 and the second one .1, which one should we focus on?

Comment author: Rain 06 July 2011 12:39:17PM *  2 points [-]

Why do I feel like there's massively more evidence than "a few blog posts"? I must be counting information I've gained from other studies, like those on human history, and lumping it all under "what intelligent agents can accomplish". I'm likely counting fictional evidence, as well; I feel sort of like an early 20th century sci-fi buff must have felt about rockets to the moon. Another large part of being convinced falls under a lack of counterarguments - rather, there are plenty out there, just none that seem to have put thought into the matter.

At any rate, I'm not asking for the entire world to throw down their asteroid detection schemes or their climate mitigation strategies; that's not politically feasible, regardless of risk probabilities. I'm just asking them to increase the size of the pie by a few million, maybe as little as one billion total, to add research about AI, and to spend more money on the whole gamut of existential risk reduction as a cohesive topic of great importance.

What would you tell the first climate scientist to examine global warming, or the first to predict asteroid strikes, other than "do more research, and get others to do research as well"?

Comment author: XiXiDu 06 July 2011 02:27:19PM *  0 points [-]

What would you tell the first climate scientist to examine global warming, or the first to predict asteroid strikes, other than "do more research, and get others to do research as well"?

I have no problem with a billion dollars spend on friendly AI research. But that doesn't mean that I agree that the SIAI needs a billion dollars right now or that I agree that the current evidence is enough to tell people to stop researching cancer therapies or create educational videos about basic algebra. I don't think we know enough about risks from AI to justify such advice. I also don't think that we should all become expected utility maximizer's because we don't know enough about economics, game theory, and decision theory and especially about human nature and the nature of discovery.

Why do I feel like there's massively more evidence than "a few blog posts"?

Maybe because there is massively more evidence and I don't know about it, don't understand it, haven't taken it into account or because I am simply biased. I am not saying that you are wrong and I am right.

...those on human history, and lumping it all under "what intelligent agents can accomplish".

Shortly after human flight was invented we reached the moon. Yet human flight is not as sophisticated as bird or insect flight, it is much more inefficient, and we never reached other stars. Therefore, what I get out of this, shortly after we invent artificial general intelligence we might reach human-level intelligence and in some areas superhuman intelligence. But that doesn't mean that it will be particularly fast or efficient, or that it will be able to take over the world shortly after. Artificial general intelligence is already an inference made from what we currently believe to be true, going a step further and drawing further inferences from previous speculations, e.g. explosive recursive self-improvement, is in my opinion a very shaky business. We have no idea about the nature of discovery, if intelligence (whatever that is) is even instrumental or quickly hits diminishing returns.

In principle we could build antimatter weapons capable of destroying worlds, but in practise it is much harder to accomplish. The same seems to be the case for intelligence. It is not intelligence in and of itself that allows humans to accomplish great feats. Someone like Einstein was lucky to be born into the right circumstances, the time was ripe for great discoveries.

Another large part of being convinced falls under a lack of counterarguments - rather, there are plenty out there, just none that seem to have put thought into the matter.

Prediction: The world is going to end.

Got any counterarguments I couldn't easily dismiss?

Most of the superficially disjunctive lines of reasoning about risks from AI derive their appeal from their inherent vagueness. It's not like you don't need any assumptions to be true to get "artificial general intelligence that can undergo explosive recursive self-improvement to turn all matter in the universe into paperclips". That's a pretty complex prediction actually.

There are various different scenarios regarding the possibility and consequences of artificial general intelligence. I just don't see why the one put forth by the SIAI is more likely to be true than others. Why for example would intelligence be a single principle that, once discovered, allows us to grow superhuman intelligence overnight? Why are we going to invent artificial general intelligence quickly, rather than having to painstakingly optimize our expert systems over many centuries? Why would intelligence be effectively applicable to intelligence itself, rather than demanding the discovery of unknown unknowns due to sheer luck or the pursuit of treatments for rare diseases in cute kittens? Why would general intelligence be at all efficient compared to expert systems, maybe general intelligence demands a tradeoff between plasticity and goal-stability? I can think of dozens of possibilities within minutes, none of them leading to existential risk scenarios.

Comment author: timtyler 06 July 2011 12:21:34PM *  1 point [-]

If there was a risk that might kill us with a probability of .7 and another risk with .1 while our chance to solve the first one was .0001 and the second one .1, which one should we focus on?

To solve this problem we need to know more. As it stands, the marginal effect of investment in the problems on the probability of the problems being solved is unknown - as is the temporal relationship of the problems. Do they arise at the same time? Is there going to be time to concentrate on the second problem after solving the first one? - etc.

Comment author: timtyler 06 July 2011 12:27:04PM *  0 points [-]

I don't trust my ability to make correct probability estimates, don't trust the overall arguments and methods and don't know how to integrate that uncertainty into my estimates. It is all too vague.

Essentially, uncertainty -> wider confidence intervals and less certainty (i.e. fewer extreme probability estimates).

Comment author: hairyfigment 04 July 2011 08:11:43PM 1 point [-]

Please convince me that your Roboto Protocol could work. I don't observe politics ever producing results like the ones you seem to require.

Comment author: Sewing-Machine 04 July 2011 03:43:38PM 1 point [-]

So far I can't tell if you agree that each claim has easily more than .5 probability given the evidence, nor if you share my view that Claim A as separate from the rest has P close to 1.

The whole dispute is about your claim A. It gives lot of credence to Y's idea of where things are headed (someone is going to write a single AI that takes over the world) and none to H's (someone is going to upload some humans and make trillions of copies). Those are two very different possibilities with different consequences, and there's no reason to believe it's close to an exhaustive list of plausible scenarios.

Comment author: hairyfigment 04 July 2011 08:17:44PM 0 points [-]

Actually, I wrote Claim A in such a way that it applies to uploads. Near as I can tell H disagrees with B and/or D, or else he believes that FAI will 'emerge' naturally (e.g. from interactions between uploads).

Comment author: Sewing-Machine 04 July 2011 11:39:18PM 1 point [-]

Well, if we understand A to include ems, then your B, C and D are misleadingly worded. They all speak of a single AI. If you reworded them to allow for ems, then I guess I would agree that the dispute is located there. But I would be interested to see such a rewording--I think it would be favorable to Hanson's point of view.

It's also not clear to me that "single powerful AI or trillions of uploads" together add up to p > .5! But that might be off-topic.

Comment author: hairyfigment 05 July 2011 07:01:01AM 0 points [-]

Favorable? I don't know why you'd think that. Seems to me the charitable interpretation of Hanson's view has him thinking of ems as naturally Friendly, or near-Friendly. (My analysis didn't mention the chance of us getting FAI without working for it.)

If we get two unFriendly AIs that individually have the power to kill humanity, and if acting quickly means they don't have to negotiate with anyone else from this planet, they'll divide Earth between them. If we somehow get trillions of uFAIs with practically different goals, then of course the expected value of killing humanity goes way down. But it still sounds greater than the expected value of cooperating with us, by Hanson's analysis. And if we get one FAI out of a trillion AGIs, I think that leads to either war or a compromise like the one the Super-Happies offered the Babyeaters. We might get a one-trillionth slice of the available matter, with no more thought given to (say) the aesthetics of the Moon than we give to any random person who'd like to see his name there in green neon every night. (Still a better deal than we'd offer any one upload according to Hanson. But maybe I've misunderstood him?)

I also don't understand how you get that much memory and processing power without some designer that seems awfully close to an AI-programming AI. But as a layman I may be thinking of that in the wrong way.

Oh, and I lean towards P(genocide) somewhat under .4 without FAI theory. Right now I'm just arguing that it exceeds .05 per XiXiDu's comment. You may have misread his "5%" there.

Comment author: Sewing-Machine 05 July 2011 07:50:04AM 0 points [-]

Favorable? I don't know why you'd think that. Seems to me the charitable interpretation of Hanson's view has him thinking of ems as naturally Friendly, or near-Friendly

You would have to tell me what friendly and unfriendly means in this context. Hanson expects ems to be very numerous and very poor. I doubt he expects any one of them to have the resources available to what's usually called an fai. Is a human being running at human speeds F or UF?

If we somehow get trillions of uFAIs with practically different goals, then of course the expected value of killing humanity goes way down. But it still sounds greater than the expected value of cooperating with us, by Hanson's analysis.

I don't think the notion of "cooperating with us" is coherent. Just as the trillions of ems might have practically different goals, so might the billions of live humans.

I also don't understand how you get that much memory and processing power without some designer that seems awfully close to an AI-programming AI.

Possibly, being poor, they would not have that much memory and processing power.

Comment author: XiXiDu 04 July 2011 12:07:21PM *  1 point [-]

And if not, if you want to argue against my claims in some other way, please do so without identifying them with a more specific storyline.

That's one of the main problems I have with the whole existential risks prediction business. There is a specific storyline, it is comprised in the vagueness of your claims. If you tried to pin down a concept like 'recursive self-improvement' that supports the notion of an existential risk, you would end up with an argument that is strongly conjunctive. Most of the arguments in favor of risks from AI derive their appeal from vagueness, that doesn't mean that they are disjunctive.

Comment author: XiXiDu 04 July 2011 11:35:14AM *  1 point [-]

Given we survive long enough, we'll find a way to write a self-modifying program that has, or can develop, human-level intelligence.

How can I arrive at the belief that it is possible for an algorithm to improve itself in a way to achieve something sufficiently similar to human-level intelligence? That it is in principle possible is not a question here. But is it possible given limited resources? And if it is possible given limited resources, is it efficient enough to pose an existential risk?

The capacity for self-modification follows from 'artificial human intelligence,' but since we've just seen links to writers ignoring that fact I thought I'd state it explicitly.

Humans can learn, that is far from what is necessary to reach a level above your own, on your own. Also, how do you know that any given level of intelligence is capable of handling its own complexity effectively? Many humans are not capable of handling the complexity of the brain of a worm.

This necessarily gives the AI the potential for greater-than-human intelligence due to our known flaws.

That humans have a hard time to change their flaws might be an actual feature, a trade off between plasticity, efficiency and the necessity of goal-stability.

Given A, the intelligence would improve itself to the point where we could no longer predict its actions in any detail.

I don't think that is a reasonable assumption, see my post here. The short version: I don't think that intelligence can be applied to itself efficiently.

...the AI could escape from any box we put it in. (IIRC this excludes certain forms of encryption, but I see no remotely credible scenario in which we sufficiently encrypt every self-modifying AI forever.)

Well, even humans can persuade their guards to let them out. I agree.

...the AI could wipe out humanity if it 'wanted' to do so.

I think it is unlikely that most AI designs will not hold. I agree with the argument that any AGI that isn't made to care about humans won't care about humans. But I also think that the same argument applies for spatio-temporal scope boundaries and resource limits. Even if the AGI is not told to hold, e.g. compute as many digits of Pi as possible, I consider it an far-fetched assumption that any AGI intrinsically cares to take over the universe as fast as possible to compute as many digits of Pi as possible. Sure, if all of that are presuppositions then it will happen, but I don't see that most of all AGI designs are like that. Most that have the potential for superhuman intelligence, but who are given simple goals, will in my opinion just bob up and down as slowly as possible. This is an antiprediction, not a claim to the contrary. What makes you sure that it will be different?

Comment author: hairyfigment 04 July 2011 08:07:31PM 0 points [-]

Humans can learn, that is far from what is necessary to reach a level above your own, on your own.

Yes, you also need the ability to self-modify and the ability to take 20 or fail and keep going. But I just argued that the phrase "on your own" obscures the issue, because if one AGI has a chance to rewrite itself (and does not take over the world) then I see no realistic way to stop another from trying at some point.

Also, how do you know that any given level of intelligence is capable of handling its own complexity effectively?

I don't think I need to talk about "any given level". If humans maintain a civilization long enough (and I don't necessarily accept Eliezer's rough timetable here) we'll understand our own level well enough to produce human-strength AGI directly or indirectly. By definition, the resulting AI will have at least a chance of understanding the process that produced it, given time. (When I try to think of an exception I find myself thinking of uploads, and perhaps byzantine programs that evolved inside computers. These might in theory fail to understand all but the human-designed parts of the process. But the second example seems unlikely on reflection, as it suggests vast amounts of wasted computation. Likewise -- though I don't know how much importance to attach to this -- it seems to this layman as if biologists laugh at uploads and consider them a much harder problem than an AI with the power to program.Yet you'd need detailed knowledge of the brain's biology to make an upload.) And of course it can think faster than we do in many areas (or if it can't due to artificial restrictions, the next one can).

I don't think that intelligence can be applied to itself efficiently.

You've established inefficiency as a logical possibility (in my judgement) but don't seem to have given much argument for it. I count two sentences on your P2 that directly address the issue. And you have yet to engage with the cumulative probability argument. Note that a human-level AGI which can see problems or risks of self-modification may also see risk in avoiding it.

Even if the AGI is not told to hold, e.g. compute as many digits of Pi as possible,

If it literally has no other goals then it doesn't sound like an AGI. The phrase "potential for superhuman intelligence" sounds like it refers to a part of the program that other people could (and, in my view, will) use to create a super-intelligence by combining it with more dangerous goals.

Comment author: Giles 04 July 2011 01:31:25AM 2 points [-]

The debate seems to be about the probability of the specific "AI in a basement goes foom" scenario. There are surely other existential risk scenarios which seem more Hansonian, e.g. firms gradually replacing all of their human capital with computers and robots until there are no humans left?

Comment author: CarlShulman 07 July 2011 05:53:33PM *  0 points [-]

Hanson's arguments here don't apply to nation-states that start off with a large portion of world research resources and stronger ability to keep secrets, as we saw with nuclear weapons. He has a quite separate argument against that, which is that governments are too stupid to notice early brain emulation or AI technology and recognize that it is about to turn the world upside down.

Comment author: timtyler 07 July 2011 11:07:01PM 0 points [-]

What: even the NSA and IARPA - whose job it is?

Comment author: CarlShulman 07 July 2011 11:23:36PM *  0 points [-]

I don't buy it at that level of confidence. Robin says the Manhattan Project was an anomaly in wartime, and that past efforts to restrict the spread of technologies like encryption and supercomputers didn't work for long (I'd say the cost-benefit for secrecy here was much, much worse than for human-level AI/WBE). My reply is that a delay of 4 years, like that between the US and Soviet nuclear tests, would be a long, long time for WBE or human-level AI to drive a local intelligence explosion. Software is easier to steal, but even so.

Comment author: timtyler 08 July 2011 07:51:32AM *  0 points [-]

Your reply is focused on keeping secrets. I meant my comment to apply to the second claim - the one about governments being "too stupid". That claim might be right - but it is not obvious. Government departments focused on this sort of thing (of which there are several) will understand - and no doubt already understand. The issue is more whether the communication lines are free, whether the top military brass take thier own boffins seriously - and whether they go on to get approval from head office.

As for secrecy - the NSA has a long history of extreme secrecy. The main reason most people don't know about their secret tech projects is because their secrecy is so good. If they develop a superintelligence, I figure it will be a secret one that will probably remain chained up in their basement. They are the main reason, my graph has some probability mass already.

Comment author: wedrifid 04 July 2011 06:35:42AM *  1 point [-]

Hanson's position being essentially destroyed by Hanson via support that made no sense...

As far as I can tell Hanson does not disagree with Yudkowsky except for the probability of risks from AI. Yudkowsky says that existential risks from AI are not under 5%. Has Yudkowsky been able to support this assertion sufficiently? Hanson only needs to show that it is unreasonable to assume that the probability is larger than 5% and my personal perception is that he was able to do so.

Note that my comment (quoted) referred to the 2008 debate, which was not on that subject.

Think I am a troll or an idiot, let me know, I want to know

You do not appear to be trolling in this particular instance.

Comment author: Giles 04 July 2011 02:27:37AM -1 points [-]

Yudkowsky has a a handicap too, since presumably he feels that if he says anything too insightful, particularly concerning AI architecture, there's a small probability that insight might be used to accelerate Unfriendly AI development.

Comment author: timtyler 06 July 2011 10:28:55PM 1 point [-]

Hanson gets polite and respectful treatment for his emulation scenario. I am not convinced that is the right approach. Emulations first is a pretty crazy idea - and Hanson doesn't appear to have been advised about that a sufficiently large number of times yet.

Comment author: jsalvatier 06 July 2011 11:14:07PM *  4 points [-]

For those of us not well schooled in these matters, can you explain or link to why it's a crazy idea? I am intuitively sympathetic to Yudkowsky, but emulations first doesn't seem obviously crazy.

Comment author: timtyler 06 July 2011 11:24:18PM *  1 point [-]

My last attempt at that: Against Whole Brain Emulation.

Anissimov had a recent post on the topic of how few computing machines are bioinspired.

As I say, it looks more like a public relations exercise than anything else. The human face of machine intelligence. Yeah, right.

Comment author: p4wnc6 09 November 2011 05:03:45AM 0 points [-]

I've been trying to put together a survey paper about why uploads coming first is not at all crazy. Whether it's more likely than local-super-AI-in-a-basement, I don't know and leave that to the experts. But brain emulation, as far as we currently understand it, is very similar to the problem posed by landing on the moon circa 1962 -- a matter of scaling it up (though we certainly have much more than a decade to go for mind emulation).

Ken Hayworth, now working in the Lichtman connectomics lab at Harvard, has recently written up such a survey paper to appear in the International Journal of Machine Consciousness this coming summer. I interviewed him and reviewed a preprint of his paper, comparing it with some counter-arguments from Ian Parberry and Paul Allen.

You can find my paper here. I would very much appreciate comments or suggestions if you have them. I'm already planning another section on compression and sensitivity issues with this much data, and also perhaps on ethical questions surrounding the first uploads. This was for a mid-term project for this course, so the scope (due to time constraints, etc.) couldn't be as large or well-addressed as I had hoped.

Comment author: timtyler 03 July 2011 11:10:16PM *  -1 points [-]

Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world. He was pro, I was con.

The thesis seems pretty obviously true to me, though there is some issue over how much is "much".

Google or Facebook control a much larger fraction the world compared to farmers or industy folk from decades ago. Essentially technological progress promotes wealth inequality by providing the powerful with technology for keeping control of their wealth and power. So, we have more wealth inequality than ever - and will most likely have even more wealth inequality in the future.

Comment author: whpearson 05 July 2011 01:55:46AM 2 points [-]

Google's revenues are $29 billion and are dwarfed by such things as walmart ($258 billion) or BP ($308 billion). They are not that important yet.

Comment author: timtyler 05 July 2011 08:23:11AM *  0 points [-]

Google's revenues are $29 billion and are dwarfed by such things as walmart ($258 billion) or BP ($308 billion). They are not that important yet.

The question says to compare with the first-movers of the industrial revolution, though.

So, this isn't Google vs Walmart but rather Larry Page vs James Watt.

It is well known that wealth inequality is now at record levels. Equally well known is that some of the richest people top pioneering tech companies. The only debate I see is over what "much larger fraction" means - and that seems to be a rather pointless thing to argue about.

Comment author: whpearson 05 July 2011 10:00:43AM *  0 points [-]

As we aren't talking about first movers exactly, I can choose the governor general of the east india company, that ruled over india? It also depends what you consider actors.

Aparantly social inequality was greater in Holland in 1732

Comment author: timtyler 06 July 2011 03:09:51PM *  0 points [-]

Aparantly social inequality was greater in Holland in 1732

Here's some more comprehensive data. It looks as though I should have said that wealth inequality is now at record levels in the USA. Other countries vary considerably.

Comment author: timtyler 05 July 2011 08:36:35PM *  0 points [-]

As we aren't talking about first movers exactly, I can choose the governor general of the east india company, that ruled over india?

Were they an early mover in the first industrial revolution?

Comment author: Sewing-Machine 04 July 2011 12:16:36AM 3 points [-]

But Google and Facebook are hardly "first movers" at the scale you're talking about.

Comment author: timtyler 04 July 2011 12:46:36AM *  2 points [-]

OK - so we have to understand that being first in the tech industry isn't what is important. Often first movers get overtaken by second movers who go on to rise to power - e.g. AltaVista, Friendster, etc. Being early is important, but being first isn't necessarily enough.

This is a pretty well-established pattern. So, yes, strictly it is "early movers" that the discussion should be about. We could discuss "first movers" - but that fails to nail the main phenomenon of interest here very well, IMO.

Comment author: Sewing-Machine 04 July 2011 12:57:20AM 4 points [-]

There's something more basic about the analogy I object to. Google, Friendster, Altavista and Facebook were not founded at the beginning of a singularity.

It seems that:

  1. The first creatures to develop human-level intelligence came to dominate all other creatures.
  2. The first humans to develop agriculture did not come to dominate all other humans

I don't know how to interpret the evidence from the industrial revolution, and I don't really see a pattern. But I think the place to look for a pattern is here, not in recent tech industry history.

Comment author: asr 04 July 2011 02:22:35PM *  4 points [-]

The first creatures to develop human-level intelligence came to dominate all other creatures.

I am not so confident of this. There were a bunch of early hominids, other than H. Sap, whose intelligence is not well established. Suppose, say, it turns out that Neanderthals were 7% smarter on average than modern humans, but lost out evolutionarily because they had too much body hair. What would that imply about the importance of intelligence?

Comment author: Vaniver 05 July 2011 03:27:48AM 0 points [-]

Actually, we have strong reason to suspect Neanderthals were smarter than Cro-Magnon man, and lost out because Cro-Magnon man traded and Neanderthals didn't.

Comment author: timtyler 04 July 2011 06:56:13AM *  1 point [-]

There's something more basic about the analogy I object to. Google, Friendster, Altavista and Facebook were not founded at the beginning of a singularity.

What do you mean by that? You think Google isn't going to go on to develop machine intelligence? Surely they are among the front runners - though admittedly there are some other players in the game. This is not a case of hanging around for some future event.

  • The first creatures to develop human-level intelligence came to dominate all other creatures.
  • The first humans to develop agriculture did not come to dominate all other humans

The wording of the question was:

Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world.

It doesn't say the control is kept indefinitely. So - for instance - Sergey and Larry might die - but they will still have quickly come to control a larger fraction of the world than any farmer or industrialist.

Comment author: timtyler 06 July 2011 11:52:06PM *  0 points [-]

Hanson has made a lot of comments recently about how intellligence is poorly defined and how we don't really know what it is - e.g. 77:30 and 83:00 minutes in. I think we do now have a pretty good idea about that - thanks to the Hutter/Legg work on universal intelligence. If Hanson was more familiar with this sort of material, I rather doubt he would say the kinds of things he is currently saying.

Comment author: MatthewBaker 07 July 2011 06:44:56PM 0 points [-]

Whoever asked Robin about his opinion that social skills separated humans from chimpanzees "Can you envision a scenario where one of the computers acquired this 'Social Skill' then said to all the other computers "hey guys lets go have a revolution" " Love that comment

Comment author: timtyler 06 July 2011 09:42:54PM *  0 points [-]

73:00 - this seems to be a mis-summary by Hanson. I am pretty sure that Norvig was saying that complex models were still useful - not that simpler ones didn't even exist.

The situation is similar to that with compression. If you can compress a bit that is still useful - and it is easier to do than compressing a lot.

Comment author: timtyler 06 July 2011 06:32:40PM *  0 points [-]

It's not a brain in a box in a basement - and it's not one grand architectural insight - but I think the NSA shows how a secretive organisation can get ahead and stay ahead - if it is big and well funded enough. Otherwise, public collaboration tends to get ahead and stay ahead, along similar lines to those Robin mentions.

Google, Apple, Facebook etc. are less-extreme versions of this kind of thing, in that they keep trade secrets which give them advantages - and don't contribute all of these back to the global ecosystem. As a result they gradually stack up know-how that others lack. If they can get enough of that, then they will gradually pull ahead - if they are left to their own devices.

The issue of whether a company will eventually pull ahead is an issue which has quite a bit to do with anti-trust legislation - as I discuss in One Big Organism.

The issue of whether one government will eventually pull ahead is a bit different. There's no government-level anti-trust legislation. However expansionist governments are globally frowned upon.

I don't think there are too many other significant players besides companies and governments.

The "silver bullet" idea doesn't seem to be worth too much. As Eray says: "Every algorithm encodes a bit of intelligence". We know that advanced intelligence necessarily highly complex. You can't predict a complex world without being that complex yourself. Of course, human intelligence might be relatively simple - in which case it might only take a few leaps to get to it. The history of machine intelligence fairly strongly suggests a long gradual slog to me - but it is at least possible to argue that people have been doing it all wrong so far.