"Long-winded arguments tend to fail" is a daring section title in your 5,000 word essay :P

In general, I think the genre "collect all the arguments I can find on only one side of a controversial topic" is bound to lead to lower-quality inclusions, and that section is probably among them. I prefer the genre "collect the best models I can find of a controversial topic and try to weigh them."

Why is "The argument for AI risk has a lot of necessary pieces, therefore it's unlikely" a bad argument?

1. Any real-world prediction can be split into an arbitrary number of conjunctive pieces.
   • "It will rain tomorrow" is the prediction "it will rain over my house, and the house next to that, and the house next to that, and the house across the street, and the house next to it, and the house next to that, etc." Surely any conjunction with so many pieces is doomed to failure, right? But wait, the prediction "it won't rain tomorrow" has the same problem.
   • Could you split the claim that future AI will do good things and not bad things into lots of conjunctions?
2. It's easy to get sloppy about whether pieces are really necessary or not.
   • E.g. "The current neural net paradigm continues" isn't strictly necessary for future AI doing bad things rather than good things. But since we're just sort of making a long list, there's the temptation to just slap it onto the list and not worry about the leakage of probability-mass when it turns out not to be necessary. But if each step leaks a little, and you're making a list of a large number of steps... well, point is it's sometimes easy to fool yourself with this argument structure.
3. Figuring out conditional probabilities is hard.
   • At the start of the list, the probabilities are, while not easy, at least straightforward to understand. E.g. your item 1 is "P(we have enough training data, somehow)". This is hard to predict, but at least it's clear what you mean. But every step of the list has to be conditional on all the previous steps.
   • (This was the flaw with the argument that "it will rain tomorrow" has low probability. Once it's raining on the first 99 houses, adding the 100th house to the list doesn't move the probabilities much.)
   • So by the time you're at item 16, you're not just asking P(item 16), you're asking P(item 16, given that 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, and 15 are true), which is often an unintuitive slice of reality it's hard to estimate probabilities about.

For what it’s worth, I find that you are equivocating in a strange way between endorsing and not endorsing these arguments.

On the one hand, here in this post you called them “the best arguments” and “tell me why I’m wrong”, which sounds a lot like an endorsement. And your post title also sounds an awful lot like an endorsement.

On the other hand, in the substack text, you say at the top that you don’t have an opinion, and you state objections without stating in your own voice that you think the objections are any good. For example, “Yann LeCun argues that the need to dominate is purely a social phenomena that does not develop because of intelligence.” Well, yes, that is true, Yann LeCun does say that. But do you think it’s a good argument? If so, you should say that! (I sure don't!—See e.g. here.)

I think you should pick one or the other rather than equivocating. If you really don’t know where you stand, then you should retitle your post etc. Or if you find some of the arguments compelling, you should say that.

+1 to one of the things Charlie said in his comment, but I’d go even further:

The proposition “The current neural architecture paradigm can scale up to Artificial General Intelligence (AGI) (especially without great breakthroughs)” is not only unnecessary for the proposition “AI is an extinction threat” to be true, it’s not even clear that it’s evidence for the proposition “AI is an extinction threat”! One could make a decent case that it’s evidence against “AI is an extinction threat”! That argument would look like: “we’re gonna make AGI sooner or later, and LLMs are less dangerous than alternative AI algorithms for the following reasons …”.

As an example, Yann LeCun thinks AGI will be a different algorithm rather than LLMs, and here’s my argument that the AGI algorithm LeCun expects is actually super-dangerous. (LeCun prefers a different term to “AGI” but he’s talking about the same thing.)

I’m trying to figure where you were coming from that you brought up “The current neural architecture paradigm can scale up to Artificial General Intelligence (AGI) (especially without great breakthroughs)” as a necessary part of the argument.

One possibility is, you’re actually interested in the question of whether transformer-architecture self-supervised (etc.) AI is an extinction threat or not. If so, that’s a weirdly specific question, right? If it’s not an extinction threat, but a different AI algorithm is, that would sure be worth mentioning, right? But fine. If you’re interested in that narrow question, then I think your post should have been titled “against transformer-architecture self-supervised (etc) AI as an extinction threat” right? Related: my post here.

Another possibility is, you think that the only two options are, either (1) the current paradigm scales to AGI, or (2) AGI is impossible or centuries away. If so, I don’t know why you would think that. For example, Yann LeCun and François Chollet are both skeptical of LLMs, but separately they both think AGI (based on a non-LLM algorithm) is pretty likely in the next 20 years (source for Chollet). I’m more or less in that camp too. See also my brief comment here.

I read it, just in case there were any new, good arguments there. Just the same ones I've seen (after spending a bunch of time over years looking at arguments on both sides).  I guess it's a useful project to collect the arguments, but it seems like any real work needs to be done in looking at how well they stand up to counterarguments. Which they do not to well.

The arguments you list do refute some arguments for AI x-risk, but not the important ones. They are largely driven by motivated reasoning, commonly called "wishful thinking".  Or sometimes the motivation is "AI doomers drive me crazy, I'd really like a way to make them look stupid". I'm afraid irritating people is a great way to lose arguments even when you're right; I've experimented with this a fair amount.

As the commenter on your substack says, all of these have been refuted; the refutations have not been refuted. I keep my eyes out and potentially waste time looking at the arguments against, because I'm fascinated by the public debate process on this topic (and how it might well get us all killed, as a secondary interest).

These arguments fall into several categories.

• a) against specific arguments for AI risk, therefore not actually against AI risk overall 
  • (the complex arguments fail argument in your piece for instance -
    • The strong arguments are simple
• b) misunderstanding the arguments for AI risk 
  • (e.g., LeCun's argument in your post simply ignores the real argument for instrumental convergence creating subgoals of power-seeking) 
• c) assuming or arguing that we won't have fully general intelligence any time soon 
  • Possible, but you'd have to be super expert and have strong reasons to be anything like sure about this. The people making this argument are lacking the expertise, the arguments, or both.

This isn't to say that AI is certain to kill us, just that there are no existing arguments for it being unlikely. Since we don't know, it would be the logical thing to do to think hard about it and try to take some precautions against AGI x-risk. I think the reality is way worse than that (very good chance we'll be pushed aside by the AGI we create, and we're way behind on thinking about how to avoid that) but at the least, the sane conclusion from just the surface level arguments is "do we have enough people working on that?" not "I found an argument that gives me an excuse to go full steam ahead without thinking about risks anymore"!

My favorite from your list is the one about bad reference classes. I agree that arguing from reference classes are bad arguments. They're useful as intuition pumps, but we need to think about the AGI we'll actually build, not any historical examples, because there are none that match closely enough to give more than a vague guess. Thinking about how things will go requires actual expertise in the relevant fields- AI, psychology, game theory, etc.