LESSWRONG
LW

Comment Permalink

For those of us not well schooled in these matters, can you explain or link to why it's a crazy idea? I am intuitively sympathetic to Yudkowsky, but emulations first doesn't seem obviously crazy.

[anonymous]14y00

Ken Hayworth, now working in the Lichtman connectomics lab at Harvard, has recently written up such a survey paper to appea... (read more)

1timtyler14y

My last attempt at that: Against Whole Brain Emulation. Anissimov had a recent post on the topic of how few computing machines are bioinspired. As I say, it looks more like a public relations exercise than anything else. The human face of machine intelligence. Yeah, right.

See in context

21 Hanson Debating Yudkowsky, Jun 2011

by XiXiDu

3rd Jul 2011

1 min read

21

On Wednesday I debated my ex-co-blogger Eliezer Yudkowsky at a private Jane Street Capital event (crude audio here, from 4:45; better video here [as of July 14]).

I “won” in the sense of gaining more audience votes — the vote was 45-40 (him to me) before, and 32-33 after the debate. That makes me two for two, after my similar “win” over Bryan Caplan (42-10 before, 25-20 after). This probably says little about me, however, since contrarians usually “win” such debates.

Our topic was: Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world. He was pro, I was con. We also debated this subject here on Overcoming Bias from June to December 2008. Let me now try to summarize my current position.

[...]

It thus seems quite unlikely that one AI team could find an architectural innovation powerful enough to let it go from tiny to taking over the world within a few weeks.

Link: overcomingbias.com/2011/07/debating-yudkowsky.html

Personal Blog

21

Mentioned in

17Partial Transcript of the Hanson-Yudkowsky June 2011 Debate

New Comment

78 comments, sorted by

top scoring

Click to highlight new comments since: Today at 7:00 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-][anonymous]14y260

Hanson often takes his turn to speak this way: he summarizes Yudkowsky's last argument, in a way that at least superficially does not seem unfair or tendentious, then explains why he doesn't find it compelling, then explains why his own position is more compelling.

Yudkowsky seems to respond to Hanson's points without summarizing them first.

I find Hanson to be hugely more effective in the recording. Is it because of this? I was less sympathetic to Yudkowsky's point of view before I started listening, so it's hard for me to tell if this is an illusion.

9Douglas_Knight14y

In this recording (and to a lesser extent in the debate with Caplan), Hanson is much louder than his opponent, probably because of microphone placement. I expect that this makes him more effective, though I haven't listened to it much.

2wedrifid14y

Partly. Hanson is a far more powerful rhetorician then Eliezer. He speaks with the kind of language that is far more persuasive to the majority of people. It is only a narrow audience with particularly high standards with respect to judging "tendentious" tendencies for whom his style backfires.

5XiXiDu14y

Off-topic: How do you like the style of Nick Bostrom? Here is another video of him.

[-]Vladimir_Nesov14y140

Modifiers like "new" or "old" in titles or filenames quickly become unhelpful. It's generally better to use dates (in this case, something like "Jun 2011" would do the trick).

[-]John_Maxwell14y120

Can someone who agrees with Yudkowsky do an extended summary of Yudkowsky's position and arguments in text, the way Hanson summarized his position and arguments in text?

[-]XiXiDu14y130

What I Think, If Not Why.

[-]timtyler14y70

Now on Youtube: Yudkowsky vs Hanson on the Intelligence Explosion - Jane Street Debate 2011.

[-]timtyler14y70

72:00 - Hanson cites the AAAI white paper which rails against alarmism.

That seems more likely to be a tussle over funding. If the panic-stricken screechers make enough noise, it might adversely affect their member's funding. Something rather like that has happened before:

It was rumoured in some of the UK national press of the time that Margaret Thatcher watched Professor Fredkin being interviewed on a late night TV science programme. Fredkin explained that superintelligent machines were destined to surpass the human race in intelligence quite soon, and

... (read more)

[-]Normal_Anomaly14y70

In both of these debates, the change in "margin of victory" between people was smaller than the number of people who voted the first time but not the second. In the debate with Yudkowsky, Hanson's margin of victory went from -5 to 1, and 20 voters dropped out. In the debate with Caplan the margin went from 32 to 5--actually getting smaller--and 17 voters dropped out. I'm not sure if we can even determine a "winner" in terms of audience popularity, with that many disappearing votes. Is it normal in debates for large numbers of audience members to not vote at the end?

0Douglas_Knight14y

vs Caplan, the margin went from -32 to -5, with 7 people dropping out. Do you have some other source of numbers? If people leave early, that is an extra source of noise, but probably unbiased. If a non-voter is expressing uncertainty, that is a useful source of information, although it would be better to know how many abstain and better if they explicitly expressed uncertainty. I would expect people to be more bashful before the debate than after and am surprised that the number of votes went down. ETA: the recording of the Eliezer-Hanson debate says that this was an increase in number of undecided people.

2Normal_Anomaly14y

Oops, I typed 12 when I meant 32. But I think it's positive 32 and 5, not negative, because he says he won.

0Douglas_Knight14y

Try reading it again for the signs. Moreover, when he says that contrarian positions win, he means that they shift people towards them, not that they win outright. This is probably because most people have never heard actual arguments for the contrarian position.

[-]David Althaus14y60

My two cents:

Is it really useful to debate such a difficult and completely unintuitive concept in 90 minutes? It took me at least 20 hours pure reading time to finish the AI-foom debate, which consists of over 50 blog-posts.
Hanson uses simpler analogies than Yudkowsky and tries to explain basic concepts, whereas Yudkowsky's information-dense arguments often aim too high, especially considering the short time.

5Vaniver14y

If anybody can understand this after 90 minutes, it's the guys at Jane Street.

6JenniferRM14y

The primary impact appears to have been to increase the undecided votes which (assuming a high quality audience) implies to me that many members of the audience were somewhat irrationally polarized to begin with (as might be reasonable) and many people heard things from "the other side" during the debate that they had never heard before and updated towards 50/50 ignorance based on the debate. (Perhaps there is some wishful thinking in this model? Hard to tell.) If this model of the debate and audience is right, it suggests that no one (neither Hanson nor Eli nor anyone else I know of), has a clearly correct model for singularity issues with slots for evidence that has been tracked down and dropped into place to give a clean and clear answer. Basically, it sound to me like most very smart people who've thought about the singularity are still confused, and more of them are beginning to notice their confusion.

0MatthewBaker14y

Some people read really fast :)

[-]timtyler14y40

Legg's Is there an Elegant Universal Theory of Prediction? is somewhat relevant to parts of this discussion.

[-]aletheilia14y40

How about a LW poll regarding this issue?

(Is there some new way to make one, since the site redesign, or are we still at vote-up-down-karma-balance pattern?)

[-]Vladimir_Nesov14y40

Hanson seems to agree that if we get human-level agents that are cheap to run, this gets us a local takeover. I don't think that having cheap chimp-level agents widely available at that time overturns the advantage of gaining access to cheap human-level agents. So if we grant that the capability of AIs gets increased gradually and publicly, all that a local group needs to take over the world is make the step from chimp-level state-of-the-art agents to human-level agents before any other group does that. If chimp-level agents are not that different from hum... (read more)

[-][anonymous]14y110

I don't have an intuition for what would happen if you ran a chimp-level intelligence very fast. The ratio Yudkowsky mentioned in the recording was 2500 years of human-in-skull thinking = 8 hours of human-in-laptop thinking. Is it completely obvious that 2500 years of chimp thinking would yield nothing interesting or dangerous?

Chimps haven't accomplished much in the last 2500 years but that's at least partly because they don't pass on insights between generations. Can we stipulate 2500 years of chimp memory, too?

[-]timtyler14y10

Hanson has made a lot of comments recently about how intellligence is poorly defined and how we don't really know what it is - e.g. 77:30 and 83:00 minutes in. I think we do now have a pretty good idea about that - thanks to the Hutter/Legg work on universal intelligence. If Hanson was more familiar with this sort of material, I rather doubt he would say the kinds of things he is currently saying.

[-]MatthewBaker14y00

Whoever asked Robin about his opinion that social skills separated humans from chimpanzees "Can you envision a scenario where one of the computers acquired this 'Social Skill' then said to all the other computers "hey guys lets go have a revolution" " Love that comment

[-]timtyler14y00

73:00 - this seems to be a mis-summary by Hanson. I am pretty sure that Norvig was saying that complex models were still useful - not that simpler ones didn't even exist.

The situation is similar to that with compression. If you can compress a bit that is still useful - and it is easier to do than compressing a lot.

[-]timtyler14y00

It's not a brain in a box in a basement - and it's not one grand architectural insight - but I think the NSA shows how a secretive organisation can get ahead and stay ahead - if it is big and well funded enough. Otherwise, public collaboration tends to get ahead and stay ahead, along similar lines to those Robin mentions.

Google, Apple, Facebook etc. are less-extreme versions of this kind of thing, in that they keep trade secrets which give them advantages - and don't contribute all of these back to the global ecosystem. As a result they gradually stack u... (read more)

[-]timtyler14y-10

Hanson gets polite and respectful treatment for his emulation scenario. I am not convinced that is the right approach. Emulations first is a pretty crazy idea - and Hanson doesn't appear to have been advised about that a sufficiently large number of times yet.

7jsalvatier14y

For those of us not well schooled in these matters, can you explain or link to why it's a crazy idea? I am intuitively sympathetic to Yudkowsky, but emulations first doesn't seem obviously crazy.

1timtyler14y

0[anonymous]14y

I've been trying to put together a survey paper about why uploads coming first is not at all crazy. Whether it's more likely than local-super-AI-in-a-basement, I don't know and leave that to the experts. But brain emulation, as far as we currently understand it, is very similar to the problem posed by landing on the moon circa 1962 -- a matter of scaling it up (though we certainly have much more than a decade to go for mind emulation). Ken Hayworth, now working in the Lichtman connectomics lab at Harvard, has recently written up such a survey paper to appear in the International Journal of Machine Consciousness this coming summer. I interviewed him and reviewed a preprint of his paper, comparing it with some counter-arguments from Ian Parberry and Paul Allen. You can find my paper here. I would very much appreciate comments or suggestions if you have them. I'm already planning another section on compression and sensitivity issues with this much data, and also perhaps on ethical questions surrounding the first uploads. This was for a mid-term project for this course, so the scope (due to time constraints, etc.) couldn't be as large or well-addressed as I had hoped.

[-]timtyler14y-20

Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world. He was pro, I was con.

The thesis seems pretty obviously true to me, though there is some issue over how much is "much".

Google or Facebook control a much larger fraction the world compared to farmers or industy folk from decades ago. Essentially technological progress promotes wealth inequality by providing the powerful with technology for keeping control of their wealth and power. So, we have more wealth inequality than ever - and will most likely have even more wealth inequality in the future.

6[anonymous]14y

But Google and Facebook are hardly "first movers" at the scale you're talking about.

4timtyler14y

OK - so we have to understand that being first in the tech industry isn't what is important. Often first movers get overtaken by second movers who go on to rise to power - e.g. AltaVista, Friendster, etc. Being early is important, but being first isn't necessarily enough. This is a pretty well-established pattern. So, yes, strictly it is "early movers" that the discussion should be about. We could discuss "first movers" - but that fails to nail the main phenomenon of interest here very well, IMO.

7[anonymous]14y

There's something more basic about the analogy I object to. Google, Friendster, Altavista and Facebook were not founded at the beginning of a singularity. It seems that: 1. The first creatures to develop human-level intelligence came to dominate all other creatures. 2. The first humans to develop agriculture did not come to dominate all other humans I don't know how to interpret the evidence from the industrial revolution, and I don't really see a pattern. But I think the place to look for a pattern is here, not in recent tech industry history.

6asr14y

I am not so confident of this. There were a bunch of early hominids, other than H. Sap, whose intelligence is not well established. Suppose, say, it turns out that Neanderthals were 7% smarter on average than modern humans, but lost out evolutionarily because they had too much body hair. What would that imply about the importance of intelligence?

0Vaniver14y

Actually, we have strong reason to suspect Neanderthals were smarter than Cro-Magnon man, and lost out because Cro-Magnon man traded and Neanderthals didn't.

1timtyler14y

What do you mean by that? You think Google isn't going to go on to develop machine intelligence? Surely they are among the front runners - though admittedly there are some other players in the game. This is not a case of hanging around for some future event. The wording of the question was: It doesn't say the control is kept indefinitely. So - for instance - Sergey and Larry might die - but they will still have quickly come to control a larger fraction of the world than any farmer or industrialist.

3[anonymous]14y

Google's revenues are $29 billion and are dwarfed by such things as walmart ($258 billion) or BP ($308 billion). They are not that important yet.

0timtyler14y

The question says to compare with the first-movers of the industrial revolution, though. So, this isn't Google vs Walmart but rather Larry Page vs James Watt. It is well known that wealth inequality is now at record levels. Equally well known is that some of the richest people top pioneering tech companies. The only debate I see is over what "much larger fraction" means - and that seems to be a rather pointless thing to argue about.

0[anonymous]14y

As we aren't talking about first movers exactly, I can choose the governor general of the east india company, that ruled over india? It also depends what you consider actors. Aparantly social inequality was greater in Holland in 1732

0timtyler14y

Here's some more comprehensive data. It looks as though I should have said that wealth inequality is now at record levels in the USA. Other countries vary considerably.

0timtyler14y

Were they an early mover in the first industrial revolution?

[-]wedrifid14y-40

Hanson's debating success is all the more impressive given that he was fighting with a handicap. Imagine how potent his debating would be if he was actually arguing for a correct position!

4AlexMennen14y

When the judges are human, defending an incorrect position is not much of a handicap in debates on issues to which human intuition is poorly attuned.

2Manfred14y

lol. Anti-downvoted.

1XiXiDu14y

He will probably start arguing for the correct position once it is supported enough as not to be destroyed by an incorrect position.

3wedrifid14y

The main thing I can recall from from the 2008 debate mentioned was Hanson's position being essentially destroyed by Hanson via support that made no sense, making Eliezer largely redundant.

1XiXiDu14y

As far as I can tell Hanson does not disagree with Yudkowsky except for the probability of risks from AI. Yudkowsky says that existential risks from AI are not under 5%. Has Yudkowsky been able to support this assertion sufficiently? Hanson only needs to show that it is unreasonable to assume that the probability is larger than 5% and my personal perception is that he was able to do so. I have already posted various arguments for why I believe that the case for risks from AI, and especially recursive self-improvement (explosive), is not as strongly supported as some people seem to think. I haven't come across a good refutation, or a strong argument to the contrary. You are of course free to suspect and declare that I was already given sufficient evidence and that my motives are not allowing me to admit that I am wrong. I won't perceive anything like that as a personal attack and ask everyone not to downvote you for any such statements. Think I am a troll or an idiot, let me know, I want to know :-)

4Giles14y

The debate seems to be about the probability of the specific "AI in a basement goes foom" scenario. There are surely other existential risk scenarios which seem more Hansonian, e.g. firms gradually replacing all of their human capital with computers and robots until there are no humans left?

0CarlShulman14y

Hanson's arguments here don't apply to nation-states that start off with a large portion of world research resources and stronger ability to keep secrets, as we saw with nuclear weapons. He has a quite separate argument against that, which is that governments are too stupid to notice early brain emulation or AI technology and recognize that it is about to turn the world upside down.

0timtyler14y

What: even the NSA and IARPA - whose job it is?

0CarlShulman14y

I don't buy it at that level of confidence. Robin says the Manhattan Project was an anomaly in wartime, and that past efforts to restrict the spread of technologies like encryption and supercomputers didn't work for long (I'd say the cost-benefit for secrecy here was much, much worse than for human-level AI/WBE). My reply is that a delay of 4 years, like that between the US and Soviet nuclear tests, would be a long, long time for WBE or human-level AI to drive a local intelligence explosion. Software is easier to steal, but even so.

0timtyler14y

Your reply is focused on keeping secrets. I meant my comment to apply to the second claim - the one about governments being "too stupid". That claim might be right - but it is not obvious. Government departments focused on this sort of thing (of which there are several) will understand - and no doubt already understand. The issue is more whether the communication lines are free, whether the top military brass take thier own boffins seriously - and whether they go on to get approval from head office. As for secrecy - the NSA has a long history of extreme secrecy. The main reason most people don't know about their secret tech projects is because their secrecy is so good. If they develop a superintelligence, I figure it will be a secret one that will probably remain chained up in their basement. They are the main reason, my graph has some probability mass already.

3hairyfigment14y

A while back I posted this minimalist account of Eliezer's case for the importance of FAI to human survival. (Claim B technically seems too specific if you want to talk about the existential risk as a whole, but I think it reflects his view.) So far I can't tell if you agree that each claim has easily more than .5 probability given the evidence, nor if you share my view that Claim A as separate from the rest has P close to 1. In particular, you said here that you believe: By the same principle, the speed or slowness of FOOM doesn't matter in the long run unless some force with the power to stop it does so, and unless this happens every single time someone creates an unFriendly AI with the power to self-modify. I have almost no confidence that humanity in general will learn from past mistakes (and precious little confidence in the subset that could write the second or third AGI). So I think we need to look at the cumulative chance for Claim B, Claim C, and perhaps even D. Even so, it seems possible that the actual risk stays within 5%. Maybe you think some form of FAI, such as Friendly uploads, will prove easy once we get the capacity for some form of AGI. Maybe you think we seem likely to kill ourselves with X before then. Maybe you think some other force(s) will stop each and every AGI. If so, I'd like to hear your reasoning. And if not, if you want to argue against my claims in some other way, please do so without identifying them with a more specific storyline. ETA: I apparently forgot how to use links. I believe this means I should go eat or sleep. Take that as you will.

1[anonymous]14y

The whole dispute is about your claim A. It gives lot of credence to Y's idea of where things are headed (someone is going to write a single AI that takes over the world) and none to H's (someone is going to upload some humans and make trillions of copies). Those are two very different possibilities with different consequences, and there's no reason to believe it's close to an exhaustive list of plausible scenarios.

0hairyfigment14y

Actually, I wrote Claim A in such a way that it applies to uploads. Near as I can tell H disagrees with B and/or D, or else he believes that FAI will 'emerge' naturally (e.g. from interactions between uploads).

1[anonymous]14y

Well, if we understand A to include ems, then your B, C and D are misleadingly worded. They all speak of a single AI. If you reworded them to allow for ems, then I guess I would agree that the dispute is located there. But I would be interested to see such a rewording--I think it would be favorable to Hanson's point of view. It's also not clear to me that "single powerful AI or trillions of uploads" together add up to p > .5! But that might be off-topic.

-1hairyfigment14y

Favorable? I don't know why you'd think that. Seems to me the charitable interpretation of Hanson's view has him thinking of ems as naturally Friendly, or near-Friendly. (My analysis didn't mention the chance of us getting FAI without working for it.) If we get two unFriendly AIs that individually have the power to kill humanity, and if acting quickly means they don't have to negotiate with anyone else from this planet, they'll divide Earth between them. If we somehow get trillions of uFAIs with practically different goals, then of course the expected value of killing humanity goes way down. But it still sounds greater than the expected value of cooperating with us, by Hanson's analysis. And if we get one FAI out of a trillion AGIs, I think that leads to either war or a compromise like the one the Super-Happies offered the Babyeaters. We might get a one-trillionth slice of the available matter, with no more thought given to (say) the aesthetics of the Moon than we give to any random person who'd like to see his name there in green neon every night. (Still a better deal than we'd offer any one upload according to Hanson. But maybe I've misunderstood him?) I also don't understand how you get that much memory and processing power without some designer that seems awfully close to an AI-programming AI. But as a layman I may be thinking of that in the wrong way. Oh, and I lean towards P(genocide) somewhat under .4 without FAI theory. Right now I'm just arguing that it exceeds .05 per XiXiDu's comment. You may have misread his "5%" there.

0[anonymous]14y

You would have to tell me what friendly and unfriendly means in this context. Hanson expects ems to be very numerous and very poor. I doubt he expects any one of them to have the resources available to what's usually called an fai. Is a human being running at human speeds F or UF? I don't think the notion of "cooperating with us" is coherent. Just as the trillions of ems might have practically different goals, so might the billions of live humans. Possibly, being poor, they would not have that much memory and processing power.

0hairyfigment14y

Taking the last part first for context: this layman thinks that just simulating a conscious brain (experiencing something other than pure terror or slow insanity) would take a lot of resources using the copy-an-airplane-with-bullet-holes approach where you don't know what the parts actually do, at least not well enough to make a self-reflective programming AI from scratch. As to the rest, I'm assuming my previous claims hold for the case of a single AGI because you seemed to argue that simply introducing a lot more AGIs changes the argument. ("Cooperating with us" therefore means not killing us all.) I started out by granting that the nature of the AIs could make a big difference. The number seems almost irrelevant. It seems like you're arguing for the possibility that no single em would have enough resources to produce super-intelligence (making assumptions about what that requires), since they might find themselves sharing a medium with trillions of competing ems before they get that far. But this appears to mean that giving more resources to any one of them (or any group with consistent goals) could easily produce super-intelligence. Someone would eventually do this. Indeed, Hanson seems to argue that a workforce of ems would help produce better technology and thus better ems. I do have to address the possibility that the normal ems themselves could stop a self-modifying AI because they would think faster than ordinary humans. That situation would certainly decrease the risk of killing humanity. But again, for that to make sense you have to assume that effective self-modification requires vast resources (or you just get trillions of self-modifiers by assumption). You may also need to assume that a super-intelligence needs even more resources to work out a plan for killing us -- otherwise the rest of the ems would seemingly have no way to discover the plan before it went into motion, except by chance. (A superior intelligence would try to include their later act

1XiXiDu14y

That's one of the main problems I have with the whole existential risks prediction business. There is a specific storyline, it is comprised in the vagueness of your claims. If you tried to pin down a concept like 'recursive self-improvement' that supports the notion of an existential risk, you would end up with an argument that is strongly conjunctive. Most of the arguments in favor of risks from AI derive their appeal from vagueness, that doesn't mean that they are disjunctive.

1XiXiDu14y

Yes, I said that I believe that even sub-human level AI pose an existential risk. At the same time I am highly skeptical of FOOM. So why don't I agree with Eliezer outright anyway? Because the risks from AI that I perceive to be a possibility are not something you can solve by inventing provable "friendliness". How are you going to make a sophisticated monitoring system friendly? Why would people want to make it friendly? How are you going to make a virus with sub-human level general intelligence friendly? Why would one do that? Risks from AI are a broad category that need meta-solutions that involve preemptive political and security measures. You need to make sure that the first intelligent surveillance systems are employed transparently and democratically so that everyone can monitor the world for the various risks ahead. We need a global immune system that keeps care that nowhere someone gets ahead of everyone else.

3Rain14y

Have you taken your own survey and published the results somewhere? Or is it only for AI researchers? It seems like there are a great deal of hidden assumptions on all sides which make these discussions go off the tracks very quickly. Some kind of basic survey with standard probability estimates might easily show where views differ. I agree that friendliness is a long shot. If you know of a better solution, please let me know. By developing a theory of friendliness and implementing it in software. Because unfriendly things are bad. By developing a theory of friendliness and implementing it in software. Because unfriendly things are bad. Sounds like a job for CEV and a friendly AI to run it on.

1XiXiDu14y

Yes I have done so. But I don't trust my ability to make correct probability estimates, don't trust the overall arguments and methods and don't know how to integrate that uncertainty into my estimates. It is all too vague. There sure are a lot of convincing arguments in favor of risks from AI. But do arguments suffice? Nobody is an expert when it comes to intelligence. Even worse, I don't think anybody knows much about artificial general intelligence. My problem is that I fear that some convincing blog posts are simply not enough. Just imagine all there was to climate change was someone with a blog who never studied the climate but instead wrote some essays about how it might be physical possible for humans to cause a global warming. Not enough, the same person then goes on to make further inferences based on the implications of those speculations. Am I going to tell everyone to stop emitting CO2 because of that? Hardly! Or imagine that all there was to the possibility of asteroid strikes was someone who argued that there might be big chunks of rocks out there which might fall down on our heads and kill us all, inductively based on the fact that the Earth and the moon are also a big rocks. Would I be willing to launch a billion dollar asteroid deflection program solely based on such speculations? I don't think so. Luckily, in both cases, we got a lot more than some convincing arguments in support of those risks. Another example: If there were no studies about the safety of high energy physics experiments then I might assign a 20% chance of a powerful particle accelerator destroying the universe based on some convincing arguments put forth on a blog by someone who never studied high energy physics. We know that such an estimate would be wrong by many orders of magnitude. Yet the reason for being wrong would largely be a result of my inability to make correct probability estimates, the result of vagueness or a failure of the methods I employed. The reason for being

4Rain14y

Why do I feel like there's massively more evidence than "a few blog posts"? I must be counting information I've gained from other studies, like those on human history, and lumping it all under "what intelligent agents can accomplish". I'm likely counting fictional evidence, as well; I feel sort of like an early 20th century sci-fi buff must have felt about rockets to the moon. Another large part of being convinced falls under a lack of counterarguments - rather, there are plenty out there, just none that seem to have put thought into the matter. At any rate, I'm not asking for the entire world to throw down their asteroid detection schemes or their climate mitigation strategies; that's not politically feasible, regardless of risk probabilities. I'm just asking them to increase the size of the pie by a few million, maybe as little as one billion total, to add research about AI, and to spend more money on the whole gamut of existential risk reduction as a cohesive topic of great importance. What would you tell the first climate scientist to examine global warming, or the first to predict asteroid strikes, other than "do more research, and get others to do research as well"?

-1XiXiDu14y

I have no problem with a billion dollars spend on friendly AI research. But that doesn't mean that I agree that the SIAI needs a billion dollars right now or that I agree that the current evidence is enough to tell people to stop researching cancer therapies or create educational videos about basic algebra. I don't think we know enough about risks from AI to justify such advice. I also don't think that we should all become expected utility maximizer's because we don't know enough about economics, game theory, and decision theory and especially about human nature and the nature of discovery. Maybe because there is massively more evidence and I don't know about it, don't understand it, haven't taken it into account or because I am simply biased. I am not saying that you are wrong and I am right. Shortly after human flight was invented we reached the moon. Yet human flight is not as sophisticated as bird or insect flight, it is much more inefficient, and we never reached other stars. Therefore, what I get out of this, shortly after we invent artificial general intelligence we might reach human-level intelligence and in some areas superhuman intelligence. But that doesn't mean that it will be particularly fast or efficient, or that it will be able to take over the world shortly after. Artificial general intelligence is already an inference made from what we currently believe to be true, going a step further and drawing further inferences from previous speculations, e.g. explosive recursive self-improvement, is in my opinion a very shaky business. We have no idea about the nature of discovery, if intelligence (whatever that is) is even instrumental or quickly hits diminishing returns. In principle we could build antimatter weapons capable of destroying worlds, but in practise it is much harder to accomplish. The same seems to be the case for intelligence. It is not intelligence in and of itself that allows humans to accomplish great feats. Someone like Einstein was lu

3Rain14y

This is the part I'd like to focus on. Restating that position from my understanding, you are unconvinced that SIAI is important to fund, and you will not pay them to convince you, and it would be perfectly fine for other people to fund them, and you will be following the area to see if they provide convincing things in the future. Is that a fair characterization?

1XiXiDu14y

Almost, I think it is important that the SIAI continues to receive at least as much as it did last year. If the SIAI's sustainability was at stake I would contribute money, I just don't know how much. I would probably devote some time to think about the whole issue, more thoroughly than I have until now. Which also hints at a general problem, I think many people lack the initial incentive that is necessary to take the whole topic seriously in the first place, seriously enough to even invest the required time and resources to analyze the available data sufficiently. I recently hinted at some problems that need to be addressed in order to convince me that the SIAI needs more money. I am currently waiting for the "exciting developments", that have been mentioned in the subsequent comments thread, to take place. Another problem is the secretive approach the SIAI seems to subscribe to. I am not convinced that a secretive approach is the right thing to do. I also don't have enough confidence to just take their word for it if they say that they are making progress. They have to figure out how to convince people that actual progress is being made, or at least attempted, without revealing too much detail. They also have to explain if they suspect Eliezer Yudkowsky to be able to solve friendly AI on his own, or otherwise how they are going to guarantee the "friendliness" of future employees. I think this thread started by timtyler is more representative of the opinion of most people (if they knew about the SIAI) than those members of lesswrong who are already sold. People here seem overly confident in what they are told without asking for further evidence. Not that I care about the AI box experiment, even prison guards can be persuaded by humans to let them out of the jail. But as timtyler said, the secretive approach employed by the SIAI, "don't ask don't tell", isn't going to convince many people any time soon. I doubt actual researchers would just trust the SIAI if they

0CarlShulman14y

How do you mean? Human planes are faster and can transport freight better. They can even self-pilot with modern AI software. The biggest weaknesses would seem to be a lack of self-reproduction and self-repair, but those aren't really part of flight.

0XiXiDu14y

Energy efficiency and maneuverability. I suppose a dragonfly would have been a better example. We never really went straight from no artificial flight towards generally superbird/insect flight. All we got are expert flight systems, no general flight systems. Even if we were handed the design for a perfect artificial dragonfly, minus the design for the flight of a dragonfly, we wouldn't be able to build a dragonfly that can take over the world of dragonflies, all else equal, by means of superior flight characteristics.

0CarlShulman14y

Where are your figures for energy efficiency? (Recalling that the comparison should be for the same speed, or energy per kilogram transported for a kilometer given the optimal speed tradeoff).

0XiXiDu14y

A Harpy Eagle can lift more than three-quarters of its body weight while the Boeing 747 Large Cargo Freighter has a maximum take-off weight of almost double its operating empty weight. I suspect that insects can do better. But my whole point is that we never reached artificial flight that is strongly above the level of natural flight. An eagle can after all catch its cargo under various circumstances like the slope of a mountain or from under the surface of water, thanks to its superior maneuverability.

2timtyler14y

To solve this problem we need to know more. As it stands, the marginal effect of investment in the problems on the probability of the problems being solved is unknown - as is the temporal relationship of the problems. Do they arise at the same time? Is there going to be time to concentrate on the second problem after solving the first one? - etc.

0timtyler14y

Essentially, uncertainty -> wider confidence intervals and less certainty (i.e. fewer extreme probability estimates).

2hairyfigment14y

Please convince me that your Roboto Protocol could work. I don't observe politics ever producing results like the ones you seem to require.

1XiXiDu14y

How can I arrive at the belief that it is possible for an algorithm to improve itself in a way to achieve something sufficiently similar to human-level intelligence? That it is in principle possible is not a question here. But is it possible given limited resources? And if it is possible given limited resources, is it efficient enough to pose an existential risk? Humans can learn, that is far from what is necessary to reach a level above your own, on your own. Also, how do you know that any given level of intelligence is capable of handling its own complexity effectively? Many humans are not capable of handling the complexity of the brain of a worm. That humans have a hard time to change their flaws might be an actual feature, a trade off between plasticity, efficiency and the necessity of goal-stability. I don't think that is a reasonable assumption, see my post here. The short version: I don't think that intelligence can be applied to itself efficiently. Well, even humans can persuade their guards to let them out. I agree. I think it is unlikely that most AI designs will not hold. I agree with the argument that any AGI that isn't made to care about humans won't care about humans. But I also think that the same argument applies for spatio-temporal scope boundaries and resource limits. Even if the AGI is not told to hold, e.g. compute as many digits of Pi as possible, I consider it an far-fetched assumption that any AGI intrinsically cares to take over the universe as fast as possible to compute as many digits of Pi as possible. Sure, if all of that are presuppositions then it will happen, but I don't see that most of all AGI designs are like that. Most that have the potential for superhuman intelligence, but who are given simple goals, will in my opinion just bob up and down as slowly as possible. This is an antiprediction, not a claim to the contrary. What makes you sure that it will be different?

0hairyfigment14y

Yes, you also need the ability to self-modify and the ability to take 20 or fail and keep going. But I just argued that the phrase "on your own" obscures the issue, because if one AGI has a chance to rewrite itself (and does not take over the world) then I see no realistic way to stop another from trying at some point. I don't think I need to talk about "any given level". If humans maintain a civilization long enough (and I don't necessarily accept Eliezer's rough timetable here) we'll understand our own level well enough to produce human-strength AGI directly or indirectly. By definition, the resulting AI will have at least a chance of understanding the process that produced it, given time. (When I try to think of an exception I find myself thinking of uploads, and perhaps byzantine programs that evolved inside computers. These might in theory fail to understand all but the human-designed parts of the process. But the second example seems unlikely on reflection, as it suggests vast amounts of wasted computation. Likewise -- though I don't know how much importance to attach to this -- it seems to this layman as if biologists laugh at uploads and consider them a much harder problem than an AI with the power to program.Yet you'd need detailed knowledge of the brain's biology to make an upload.) And of course it can think faster than we do in many areas (or if it can't due to artificial restrictions, the next one can). You've established inefficiency as a logical possibility (in my judgement) but don't seem to have given much argument for it. I count two sentences on your P2 that directly address the issue. And you have yet to engage with the cumulative probability argument. Note that a human-level AGI which can see problems or risks of self-modification may also see risk in avoiding it. If it literally has no other goals then it doesn't sound like an AGI. The phrase "potential for superhuman intelligence" sounds like it refers to a part of the program that other pe

1wedrifid14y

Note that my comment (quoted) referred to the 2008 debate, which was not on that subject. You do not appear to be trolling in this particular instance.

-2Giles14y

Yudkowsky has a a handicap too, since presumably he feels that if he says anything too insightful, particularly concerning AI architecture, there's a small probability that insight might be used to accelerate Unfriendly AI development.

Moderation Log