Here's my serious claim after giving this an awful lot of thought and study: you are correct about the arguments for doom being either incomplete or bad.

But the arguments for survival are equally incomplete and bad.

It's like arguing about whether humans would survive driving a speedboat at ninety miles an hour before anyone's invented the internal combustion engine. There are some decent arguments that they wouldn't, and some decent argument that their engineering would be up to the challenge. People could've debated endlessly.

The correct answer is that humans often survive driving speedboats at ninety miles and hour and sometimes die doing it. It depends on the quality of engineering and the wisdom of the pilot (and how their motivations are shaped by competition;).

So, to make progress on actual good arguments, you have to talk about specifics: what type of AGI we'll create, who will be in charge of creating it and perhaps directing it, and exactly what strategies we'll use to ensure it does what we want and that humanity survives. (That is what my work focuses on.)

In the absence of doing detailed reasoning about specific scenarios, you're basically taking a guess. It's no wonder ... (read more)

A strong good argument has the following properties:

• it is logically simple (can be stated in a sentence or two)
  • This is important, because the longer your argument, the more details that have to be true, and the more likely that you have made a mistake.  Outside the realm of pure-mathematics, it is rare for an argument that chains together multiple "therefore"s to not get swamped by the fact that

No, this is obviously wrong.

1. Argument length is substantially a function of shared premises.  I would need many more sentences to convey a novel argument about AI x-risk to someone who had never thought about the subject before, than to someone who has spent a lot of time in the field, because in all likelihood I would first need to communicate and justify many of the foundational concepts that we take for granted.
   1. Note that even here, on LessWrong, this kind of detailed argumentation is necessary to ward off misunderstandings.  
2. Argument strength is not an inverse function with respect to argument length, because not every additional "piece" of an argument is a logical conjunction which, if false, renders the entire argument false.  Many details in any specific argument are

Over what timeframe?  2-20% seems a reasonable range to me, and I would not call it "very low".  I'm not sure there is a true consensus, even around the LW frequent posters, but maybe I'm wrong and it is very low in some circles, though it's not in the group I watch most.   It seems plenty high to motivate behaviors or actions you see as influencing it.

Doom aside, do you expect AI to be smarter than humans? If so, do you nonetheless expect humans to still control the world? 

Right now, every powerful intelligence (e.g. nation-states) is built out of humans, so the only way for such organizations to thrive is to make sure the constituent humans thrive, for instance by ensuring food, clean air and access to accurate information.

AI is going to loosen up this default pull. If we are limited to reflex-based tool AIs like current LLMs, probably we'll make it through just fine, but if we start doing wild adversarial searches that combine tons of the tool-like activities into something very powerful and autonomous, these can determine ~everything about the world. Unless all winners of such searches actively promote human thriving in their search instead of just getting rid of humanity or exploiting us for raw resources, we're doomed.

There's lots of places where we'd expect adversarial searches to be incentivized, most notably:

• War/national security
• Crime and law enforcement
• Propaganda to sway elections
• Market share acquisition for companies (not just in advertising but also in undermining competitors)

The current situation for war/national security is already super precarious due to nukes, and I tend to reason by an assumption that if a nuke is used again th... (read more)

I start with a very low prior of AGI doom ... I give 1 bit of evidence collectively for the weak arguments

Imagine that you have to argue with someone who believes in 50% doom^[1] on priors. Then you'd need to articulate the reasons for adopting your priors. Priors are not arguments, it's hard to put them into arguments. Sometimes priors are formed by exposure to arguments, but even then they are not necessarily captured by those arguments. Articulation of your priors might then look similarly tangential and ultimately unconvincing to those who don't shar... (read more)

As an aside, most arguments for almost anything are bad or weak, whether the conclusion is true/real or not. Science, politics, economics, really any field where there's room for uncertainty and a lot of people interested in the answer. As such, this is not strong evidence in and of itself. One sufficiently strong argument can outweigh all the bad ones. At least in terms of logical evidence. There are many, many, many people who understand your points about nuclear power, for example, but they have been unable to sway political processes for the past few d... (read more)

Have you read the sequences? My response depends on whether you have or haven't.

Here's an entirely separate weak argument, improving on your straw man:

AGI will be powerful. Powerful agentic things do whatever they want. People will try to make AGI do what they want. They might succeed or fail. Nobody has tried doing this before, so we have to guess what the odds of their succeeding are. We should notice that they won't get many second chances because agents want to keep doing what they want to do. And notice that humans have screwed up big projects in surprisingly dumb (in retrospect) ways.

If some people do succeed at making AGI do wh... (read more)

I basically only believe the standard "weak argument" you point at here, and that puts my probability of doom given strong AI at 10-90% ("radical uncertainty" might be more appropriate).

It would indeed seem to me that either I) you are using the wrong base-rate or 2) you are making unreasonably weak updates given the observation that people are currently building AI, and it turns out it's not that hard.

I'm personally also radically uncertain about correct base rates (given that we're now building AI) so I don't have a strong argument for why yours is wrong. But my guess is your argument for why yours is right doesn't hold up.

I start with a very low prior of AGI doom (for the purpose of this discussion, assume I defer to consensus).

You link to a prediction market (Manifold's "Will AI wipe out humanity before the year 2100", curretly at 13%).

Problems I see with using it for this question, in random order:

1. It ends in 2100 so the incentive is effectively about what people will believe a few years from now, not about the question. It is a Keynesian beauty contest. (Better than nothing.)
2. Even with the stated question, you win only if it resolves NO, so it is strategically correct to b

Note that some of the best arguments are of the shape "AI will cause doom because it's not that hard to build the following..." followed by insights about how to build an AI that causes doom. Those arguments are best rederived privately rather than shared publicly, and by asking publicly you're filtering the strength of arguments you might get exposed to.