LESSWRONG
is fundraising!
LW
$

by joshc
1 min read
A
411

6

If you could link to an article or other piece of media, that would be ideal. Writing one up here is fine as well. An equivalent question would be "what is the best argument for the claim that there is a <1% probability of AI existential risk?"

AI
Frontpage

6

What is the best critique of AI existential risk arguments?
New Answer
New Comment

4 Answers sorted by
top scoring

JBlack

127

Less than 1%? You need to go to fairly ridiculous extremes to get that level of certainty for even things that we know a lot about. I think that level of certainty for any such question is beyond any rational argument with current evidence.

There are plenty of arguments that would work, if only you held some particular prior belief to that level. A rock-solid belief in a higher power that would prevent it would suffice - for those who believe in such a thing at a 99%+ level. A similarly strong belief that AGI is actually impossible and not just difficult would also suffice.

[-]Vladimir_Nesov20

There are many good arguments, but not that particular "<1% probability" proof that the question requests. All the good arguments rely on uncertain assumptions, don't reach the requisite standard of proof, especially when considered together with the assumptions.

So by answering this way you are steelmanning the question (which it desperately needs).

Reply

Thomas Larsen

10

An anonymous academic wrote a review of Joe Carlsmith's 'Is power seeking AI an existential risk?', in which the reviewer assigns for <1/100,000 probability of AI existential risk. The arguments given aren't very good imo, but maybe worth reading. 

Phil Tanny

1-2

If we were to respond specifically to the title of the post....

What is the best critique of AI existential risk arguments?

I would cast my vote for the premise that AI  risk arguments don't really matter so long as a knowledge explosion feeding back upon itself is generating ever more, ever larger powers, at an ever accelerating rate.

For example, let's assume for the moment that 1) AI is an existential risk, and 2) we solve that problem somehow so that AI becomes perfectly safe.  Why would that matter if civilization is then crushed when we lose control of some other power emerging from the knowledge explosion?   Remember, triumphing over existential risk will require us to win every single time, and never lose once.  

If it's true that 1) the knowledge explosion is accelerating, and if it's true that 2) human ability is limited, then it follows that at some point we will be overwhelmed by one or more challenges that we can't adapt to in time.

Seventy five years after Hiroshima we still have no idea what to do about nuclear weapons, nor do we know what to do about AI, or genetic engineering.  And the threats keep coming, more and more, larger and larger, faster and faster.

If it is our choice to accept an ever accelerating knowledge explosion as a given, the best critique of AI existential risk arguments seems to be that they don't really matter.   Or, if you prefer, that they are a distraction from what does matter.

[-]Mitchell_Porter30

Your paradigm, if I understand it correctly, is that the self-sustaining knowledge explosion of modern times is constantly hatching new technological dangers, and that there needs to be some new kind of response - from the whole of civilization? just from the intelligentsia? It's unclear to me if you think you already have a solution. 

You're also saying that focus on AI safety is a mistake, compared with focus on this larger recurring process, of dangerous new technologies emerging thanks to the process of discovery. 

There are in fact good argume... (read more)

Reply
1Phil Tanny
Thanks much for your engagement Mitchell, appreciated. Yes,  to quibble just a bit, not just self sustaining, but also accelerating.   The way I often put it is that we need to adapt to the new environment created by the success of the knowledge explosion.  I just put up an article on the forum which explains further: https://www.lesswrong.com/posts/nE4fu7XHc93P9Bj75/our-relationship-with-knowledge As I imagine it, the needed adaptation would start with intellectual elites, but eventually some critical mass of the broader society would have to agree, to some degree or another.  I've been writing about his for years now, and can't actually provide any evidence that intellectual elites can lead on this, but who else? I don't have a ten point plan or anything, just trying to encourage this conversation where ever I go.   Success for me would be hundreds of intelligent well educated people exploring the topic in earnest together.  That is happening to some degree already, but not with the laser focus on the knowledge explosion that I would prefer. I see AI discussions as a distraction, as an addressing of symptoms, rather than addressing the source of X risks.   If 75% of the time we were discussing the source of X risk, I wouldn't object to 25% addressing particular symptoms.    I'm attempting to apply common sense.  If one has puddles all around the house every time it rains, the focus should be on fixing the hole in the roof.  Otherwise one spends the rest of one's life mopping up the puddles. I don't doubt AI can make a contribution in some areas, no argument there.   But I don't see any technology as being pivotal.  I see the human condition as being pivotal.   I'm attempting to think holistically, and consider man and machine as a single operation, with the success of that operation being dependent upon the weakest link, which I propose to be us.   Knowledge development races ahead at an ever accelerating rate, while human maturity inches along at an incre
6Mitchell_Porter
We believe AI is pivotal because we think it's going to surpass human intelligence soon. So it's not just another technology, it's our successor.  The original plan of MIRI, the AI research institute somewhat associated with this website, was to identify a value system and a software architecture for AI, that would still be human-friendly, even after it bootstrapped itself to a level completely beyond human control or understanding, becoming the metaphorical "operating system" in charge of all life on Earth.  More recently, given the rapidity of advances in the raw power of AI, they have decided that there just isn't time to solve these design problems, before some AI lab somewhere unwittingly hatches a superintelligent AI system that steamrolls the human race, not out of malice, but simply because it has goals that aren't sufficiently finetuned to respect human life, liberty, or happiness.  Instead, their current aim is to buy time for humanity, by using early superintelligent AI, to neutralize all other dangerous AI projects, and establish a temporary regime in which civilization can deliberate on what to do with the incredible promise and peril of AI and related technologies.  There is therefore some similarity with your own idea to slow things down, but in this scenario, it is to be done by force, and by using the dangerous technology of superintelligent AI, when it first appears. Continuing the operating system metaphor, this amounts to putting AI-augmented civilization into a "safe mode" before it can do anything too destructive.  This suggests a model of the future, in which there is a kind of temporary world government, equipped with a superintelligent AI that monitors everything everywhere, and which steps in to sabotage any unapproved technology that threatens to create unfriendly superintelligence. Ideally, this period lasts as long as it takes, for humanity's wise ones to figure out how to make fully autonomous superintelligence, something that we c
3 comments, sorted by
top scoring
Click to highlight new comments since:
[-]Vladimir_Nesov97

the claim that there is a <1% probability of AI existential risk

That's a serious constraint. What possible argument that's not literally a demonstration of a working AGI is going to do that to the epistemic state about a question this confusing? Imagining a future where AI is not an existential risk is easy (and there are many good arguments for it being more likely than one would expect, just as there are many good arguments for it being less likely than one would expect). But imagining a present where it's known to not be an existential risk with 99% probability (or 1% probability), despite not having already been built, doesn't work for me.

Maybe there is 0.1% probability (I sorta tried to actually assess the order of magnitude for this number) that in 15 years the world's state of knowledge builds up to a point where that epistemic state becomes thinkable (conditional on actual AGI not having been built). This would most likely require shockingly better alignment theory and expectation that less aligned AGIs can't (as in alignment-by-default) or won't be built first.

Reply
[-]Jeff Rose30

The two questions you pose are not equivalent.  There are critiques of AI existential risk arguments. Some of them are fairly strong.  I am unaware of any which do a good job of quantifying the odds of AI existential risk.  In addition, your second question appears to be asking for a cumulative probability.  It's hard to see how you can provide that absent a mechanism for eventually cutting AI existential risk to zero...which seems difficult.

Reply
Moderation Log
A
43