= 27d567bc2d48d9f40e9ccc20ea370b15)
eli_sennesh comments on International cooperation vs. AI arms race - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (143)
No, we're not talking about that kind of war. We're not talking about a balance of power that can be maintained through anti-proliferation laws (though I certainly support international agreements to not build AI and contribute to a shared, international FAI project!). If we get to the point of an American FAI versus a Chinese FAI, the two AIs will negotiate a rational compromise to best suit the American and Chinese CEVs (which won't even be that different compared to, say, Clippy).
Whereas if we get one UFAI that manages to go FOOM, it doesn't fucking matter who built it: we're all dead.
So the issue is not, "You don't build UFAI and I won't build UFAI." The issue is simply: don't build UFAI, ever, at all. All humans have rational reason to buy this proposition.
There are actually two better options here than preemptively plotting an existential-risk-grade war. They are not dichotomous and I personally support employing both.
Plot an international treaty to limit the creation of FOOM-able AIs outside a strict framework of cooperative FAI development that involves a broad scientific community and limits the resources needed for rogue states or organizations to develop UFAI. This favors the singleton approach advocated by Nick Bostrom and Eliezer Yudkowsky, and also avoids thermonuclear war. An Iraq-style conventional war of regime change is already a severe enough threat to bend most nations' interests in favor of either cooperative FAI development or just not developing AI.
For the case of a restricted-domain FAI being created, encourage global economic cooperation and cultural interaction, to ensure that whether the first FAI is Chinese or American, it will infer values over humans of a more global rather than parochial culture and orientation (though I had thought Eliezer's cognitivist approach to human ethics was meant to be difficult to corrupt using mere cultural brainwashing).
That leaves the following military options: in case of a regime showing signs of going rogue and developing their own AI, utilize conventional warfare (which in an increasingly economically interconnected world is already extremely painful for anyone except North Korea or the very poor, neither of which are good at building AIs). In case of an actual UFAI appearing and beginning a process of paper-clipping the world within a timespan that we can see it coming before it kills us: consider annihilating the planet.
However, strangely enough, none of these options suit the cultural fetish around here for sitting around in hooded cloaks plotting the doom of others in secret and feeling ever-so-"rational" about ourselves for being willing to engage in deception, secrecy, and murder for the Greater Good. So I predict people here won't actually want to take those options, because the terminal goal at work is Be Part of the Conspiracy rather than Ensure the First Superintelligence is Friendly.
Thanks, Eli. You make some good points amidst the storm. :)
I think the scenario James elaborated was meant to be a fictional portrayal of a bad outcome that we should seek to avoid. That it was pasted without context may have given the impression that he actually supported such a strategy.
I mostly agree with your bullet points. Working toward cooperation and global unification, especially before things get ugly, is what I was suggesting in the opening post.
Even if uFAI would destroy its creators, people still have incentive to skimp on safety measures in an arms-race situation because they're trading off some increased chance of winning against some increased chance of killing everyone. If winning the race is better than letting someone else win, then you're willing to tolerate some increased risk of killing everyone. This is why I suggested promoting internationalist perspective as one way to improve the situation -- because then individual countries would care less about winning the race.
BTW, it's not clear that Clippy would kill us all. Like in any other struggle for power, a newly created Clippy might compromise with humans by keeping them alive and giving them some of what they want. This is especially likely if Clippy is risk averse.
Interesting. So there are backup safety strategies. That's quite comforting to know, actually.
Oh thank God. I'd like to apologize for my behavior, but to be honest this community is oftentimes over my Poe's Law Line where I can no longer actually tell if someone is acting out a fictional parody of a certain idea or actually believes in that idea.
Next time I guess I'll just assign much more probability to the "this person is portraying a fictional hypothetical" notion.
Sorry, could you explain? I'm not seeing it. That is, I'm not seeing how increasing the probability that your victory equates with your own suicide is better than letting someone else just kill you. You're dead either way.
No worries. :-)
Say that value(you win) = +4, value(others win) = +2, value(all die) = 0. If you skimp on safety measures for yourself, you can increase your probability of winning relative to others, and this is worth some increased chance of killing everyone. Let me know if you want further clarification. :) The final endpoint of this process will be a Nash equilibrium, as discussed in "Racing to the Precipice," but what I described could be one step toward reaching that equilibrium.
Oh, how... rebel of you.
May I recommend less drama?
Frankly, when someone writes a post recommending global thermonuclear war as a possible option, that's my line. My suggested courses of action are noticeably less melodramatic and noticeably closer to the plain, boring field of WW3-prevention.
But I gave you the upvote anyway for calling out my davkanik tendencies.
I'm genuinely confused. There's an analogy to a nuclear arms race running through the OP, but as best I can tell it's mostly linking AI development controls to Cold War-era arms control efforts -- which seems reasonable, if inexact. Certainly it's not advocating tossing nukes around.
Can you point me to exactly what you're responding to?
Ah, I seem to be referring to James' excerpt from his book rather than the OP:
Oh, that makes more sense. I'd assumed, since this thread was rooted under the OP, that you were responding to that.
After reading James's post, though, I don't think it's meant to be treated as comprehensive, much less prescriptive. He seems to be giving some (fictional) outlines of outcomes that could arise in the absence of early and aggressive cooperation on AI development; the stakes at that point are high, so the consequences are rather precipitous, but this is still something to avoid rather than something to pursue. Reading between the lines, in fact, I'd say the policy implications he's gesturing towards are much the same as those you've been talking about upthread.
On the other hand, it's very early to be hashing out scenarios like this, and doing so doesn't say anything particularly good about us from a PR perspective. It's hard enough getting people to take AI seriously as a risk, full stop; we don't need to exacerbate that with wild apocalyptic fantasies just yet.
This bears investigating. I mean, come on, the popular view of AI among the masses is that All AI Is A Crapshoot, that every single time it will end in the Robot Wars. So how on Earth can it be difficult to convince people that UFAI is an issue?
I mean, hell, if I wanted to scare someone, I'd just point out that no currently-known model of AGI includes a way to explicitly specify goals desirable to humans. That oughtta scare folks.
I've talked to a number of folks who conclude that AIs will be superintelligent and therefore will naturally derive and follow the true morality (you know, the same one we do), and dismiss all that Robot Wars stuff as television crap (not unreasonably, as far as it goes).
Which one's that, eh ;-)?
Are these religious people? I mean, come on, where do you get moral realism if not from some kind of moral metaphysics?
Certainly it's not unreasonable. One UFAI versus humans with no FAI to fight back, I wouldn't call anything so one-sided a war.
(And I'm sooo not making the Dalek reference that I really want to. Someone else should do it.)
Pedantic complaint about language: moral realism simply says that moral claims do state facts, and at least some of them are true. It takes further assumptions ("internalism") to claim that these moral facts are universally compelling in the sense of moving any intelligent being to action. (I personally believe the latter assumption to be nonsense, hence AGI is a really bad idea.)
Granted, I don't know of any nice precise term for that position that all intelligent beings must necessarily do the right thing, possibly because it's so ridiculous no philosopher would profess it publicly in such words. On the other hand, motivational internalism would seem to be very intuitive, judging by the pervasiveness of the view that AI doesn't pose any risk.
I've never had that conversation with explicitly religious people, and moral realism at the "some things are just wrong and any sufficiently intelligent system will know it" level is hardly unheard of among atheists.
From abstract reason or psychological facts, or physical facts, or a mixture.
There is a subject called economics. It tells you how to achieve certain goals, such as maximising GDP. It doesn't do that by corresponding to a metaphysical Economics Object, it does that with a mixture of theoretical reasoning and examination of evidence.
There is a subject called ethics. It tells you how to achieve certain goals, such as maximising happiness....
Well, there's a couple prongs to that. For one thing, it's tagged as fiction in most people's minds, as might be suggested by the fact that it's easily described in trope. That's bad enough by itself.
Probably more importantly, though, there's a ferocious tendency to anthropomorphize this sort of thing, and you can't really grok UFAI without burning a good bit of that tendency out of your head. Sure, we ourselves aren't capital-F Friendly, but we're a far cry yet from a paperclip maximizer or even most of the subtler failures of machine ethics; a jealous or capricious machine god is bad, but we're talking Screwtape here, not Azathoth. HAL and Agent Smith are the villains of their stories, but they're human in most of the ways that count.
You may also notice that we tend to win fictional robot wars.
Also, note that the tropes tend to work against people who say "we have a systematic proof that our design of AI will be Friendly". In fact, in general the only way a fictional AI will turn out 'friendly' is if it is created entirely by accident - ANY fictional attempt to intentionally create a Friendly AI will result in an abomination, usually through some kind of "dick Genie" interpretation of its Friendliness rules.
Yeah. I think I'd consider that a form of backdoor anthropomorphization by way of vitalism, though. Since we tend to think of physically nonhuman intelligences as cognitively human, and since we tend to think of human ethics and cognition as something sacred and ineffable, fictional attempts to eff them tend to be written as crude morality plays.
Intelligence arising organically from a telephone exchange or an educational game or something doesn't trigger the same taboos.
Looks like you (emphasis mine):
and
You can be a contrarian with less drama perfectly well :-)
I would note that "we are all in the process of dying horribly" is actually a pretty dramatic situation. At the moment, actually, I'm not banking on ever seeing it: I think actual AI creation requires such expertise and has such extreme feasibility barriers that successfully building a functioning software-embodied optimization process tends to require such group efforts that someone thinks hard about what the goal system is.
Given that "we are all in the process of dying" is true for all living beings for as long as living beings existed, I don't see anything dramatic in here. As to "horribly", what is special about today's "horror" compared to, say, a hundred years ago?
I hadn't meant today. I had meant in the case of a UFAI getting loose. That's one of those rare situations where you should consider yourself assuredly dead already and start considering how you're going to kill the damn UFAI, whatever that costs you.
Whereas in the present day, I would not employ "nuke it from orbit; only way to be sure" solutions to, well, anything.
The currently fashionable descriptor is "metacontrarianism" - you might get better responses if you phrase your objection in that way.
(man, I LOVE when things go factorially N-meta)
I'm not actually sure who the metacontrarian is here.
Hence my delight in the factorial metaness.