Hanson Debating Yudkowsky, Jun 2011

XiXiDu

On Wednesday I debated my ex-co-blogger Eliezer Yudkowsky at a private Jane Street Capital event (crude audio here, from 4:45; better video here [as of July 14]).

I “won” in the sense of gaining more audience votes — the vote was 45-40 (him to me) before, and 32-33 after the debate. That makes me two for two, after my similar “win” over Bryan Caplan (42-10 before, 25-20 after). This probably says little about me, however, since contrarians usually “win” such debates.

Our topic was: Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world. He was pro, I was con. We also debated this subject here on Overcoming Bias from June to December 2008. Let me now try to summarize my current position.

[...]

It thus seems quite unlikely that one AI team could find an architectural innovation powerful enough to let it go from tiny to taking over the world within a few weeks.

Link: overcomingbias.com/2011/07/debating-yudkowsky.html

On Wednesday I debated my ex-co-blogger Eliezer Yudkowsky at a private Jane Street Capital event (crude audio here, from 4:45; better video here [as of July 14]).

I “won” in the sense of gaining more audience votes — the vote was 45-40 (him to me) before, and 32-33 after the debate. That makes me two for two, after my similar “win” over Bryan Caplan (42-10 before, 25-20 after). This probably says little about me, however, since contrarians usually “win” such debates.

Our topic was: Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world. He was pro, I was con. We also debated this subject here on Overcoming Bias from June to December 2008. Let me now try to summarize my current position.

[...]

It thus seems quite unlikely that one AI team could find an architectural innovation powerful enough to let it go from tiny to taking over the world within a few weeks.

Link: overcomingbias.com/2011/07/debating-yudkowsky.html

Favorable? I don't know why you'd think that. Seems to me the charitable interpretation of Hanson's view has him thinking of ems as naturally Friendly, or near-Friendly. (My analysis didn't mention the chance of us getting FAI without working for it.)

If we get two unFriendly AIs that individually have the power to kill humanity, and if acting quickly means they don't have to negotiate with anyone else from this planet, they'll divide Earth between them. If we somehow get trillions of uFAIs with practically different goals, then of course the expected value of killing humanity goes way down. But it still sounds greater than the expected value of cooperating with us, by Hanson's analysis. And if we get one FAI out of a trillion AGIs, I think that leads to either war or a compromise like the one the Super-Happies offered the Babyeaters. We might get a one-trillionth slice of the available matter, with no more thought given to (say) the aesthetics of the Moon than we give to any random person who'd like to see his name there in green neon every night. (Still a better deal than we'd offer any one upload according to Hanson. But maybe I've misunderstood him?)

I also don't understand how you get that much memory and processing power without some designer that seems awfully close to an AI-programming AI. But as a layman I may be thinking of that in the wrong way.

Oh, and I lean towards P(genocide) somewhat under .4 without FAI theory. Right now I'm just arguing that it exceeds .05 per XiXiDu's comment. You may have misread his "5%" there.

Favorable? I don't know why you'd think that. Seems to me the charitable interpretation of Hanson's view has him thinking of ems as naturally Friendly, or near-Friendly

You would have to tell me what friendly and unfriendly means in this context. Hanson expects ems to be very numerous and very poor. I doubt he expects any one of them to have the resources available to what's usually called an fai. Is a human being running at human speeds F or UF?

If we somehow get trillions of uFAIs with practically different goals, then of course the expected value

... (read more)

21

Hanson Debating Yudkowsky, Jun 2011

21

21

21

Hanson Debating Yudkowsky, Jun 2011

21

21