One key question is where this argument fails - because as noted, superforecasters are often very good, and most of the time, listing failure modes or listing what you need is effective.

I think the answer is adversarial domains. That is, when there is a explicit pressure to find other alternatives. The obvious place this happens is when you're actually facing a motivated opponent - like the scenario of AI trying to kill people, or cybersecurity intrusions. That's because by construction, the blocked examples don't contain much probability mass, since the opponent is actually blocked, and picks other routes. When there's an argument, the selection of arguments and the goal of the arguer is often motivated beforehand, and will pick other "routes" in the argument - and really good arguers will take advantage of this, as noted. And this is somewhat different from the Fatima Sun Miracle, where the selection pressure for proofs of God was to find examples of something they couldn't explain, and then use that, rather than selection on the arguments themselves.

In contrast, what Rethink did for theories of consciousness seems to be different - there's a priori no reason to think that most probability mass lies outside of what we think about, since how consciousness works is not understood, but is not adversarial. And moving away from the point of the post, the conclusion should be that we know we're wrong, because we haven't dissolved the question, but we can try our theories since they seem likely to be at least near the correct explanation, even if we haven't found it yet. And using heuristics, "just read the behavioural observations on different animals and go off of vibes" rather than theories, when you don't have correct theories, is a reasonable move, but also a completely different discussion!

A bunch of superforecasters were asked what their probability of an AI killing everyone was. They listed out the main ways in which an AI could kill everyone (pandemic, nuclear war, chemical weapons) and decided none of those would be particularly likely to work, for everyone.

As someone who participated in that XPT tournament, that doesn't match what I encountered. Most superforecasters didn't list those methods when they focused on AI killing people. Instead, they tried to imagine how AI could differ enough from normal technology that it could attempt to start a nuclear war, and mostly came up with zero ways in which AI could be powerful enough that they should analyze specific ways in which it might kill people.

I think Proof by Failure of Imagination describes that process better than does EFA.

I believe in Thinking Fast and Slow, Kahneman refers to this fallacy as "What You See Is All There Is" (WYSIATI). And it used to be common for people to talk about "Unknown Unknowns" (things you don't know, that you also don't know you don't know).

Non-Exhaustive Free Association or Attempted Exhaustive Free Association seems like a more accurate term?

Edit: oh ops, @Mateusz Bagiński beat me to it. convergence!

I would call this "false exhaustiveness" or "illusory exhaustiveness", else the name contains no criticism, and even mildly implies true exhaustiveness via free association (an implication invalidated by free association being usually incomplete and thus not locally valid, but I predict this matters for describing fallacies to folks not used to thinking truly exhaustively.) Also, the exhaustiveness need not be via free association. My comment is a bid for you to edit the post, because I think we should standardize on a slightly clearer name.

Claude suggested a number of existing names, but they're ones others already mentioned and don't specifically call out the illusion produced by listing many items.

It seems likely to my intuition that False Exhaustiveness is a subset of Argument from Ignorance.

edit: added missing paren, second paragraph break

This does seem to be the form of argument used when it is demanded that someone proves a negative. I'm surprised no one has brought that up elsewhere in the comments.

It is incumbent on someone making a positive statement to provide evidence. When someone doesn't provide enough evidence for a positive claim critics will point that out but "there is insufficient evidence" can only be repeated so many times.

People continue to press "Well, prove why [positive claim] isn't the case!" and the critic's only response is to iterate through all the positive arguments made for [positive claim] and show they are insufficient.

Considered as a logical fallacy, it's a form of hasty generalization.

But it's interesting to call out specifically the free-associative step; the attempt to brainstorm possibilities, nominally with the intent of checking them. That's a step where we can catch ourselves and say, "Hey, I'm doing that thing. I should be careful that I actually know what territory I'm mapping before I try to exhaustively map it out."

Going to the AI Doom example:

A bunch of superforecasters were asked what their probability of an AI killing everyone was. They listed out the main ways in which an AI could kill everyone (pandemic, nuclear war, chemical weapons) and decided none of those would be particularly likely to work, for everyone. They ended up giving some ridiculously low figure, I think it was less than one percent. Their exhaustive free association did not successfully find options like "An AI takes control of the entire supply chain, and kills us by heating the atmosphere to 150 C as a by-product of massive industrial activity."

The point at which reasoning has gone astray is in what sort of possibilities are being listed-out. The set {pandemic, nuclear war, chemical weapons} seems to be drawn from the set of known natural and man-made disasters: things that have already killed a lot of people in the past.

But the set that should be listed-out is actually all conditions that humans depend on to live. This includes a moderate-temperature atmosphere, land to grow food on, and so forth. Negating any one of those kills off the humans, regardless of whether it looks like one of those disasters we've survived before.

The forecaster is asking, "For each type of disaster, could runaway AI create a disaster of that type sufficient to kill all humans?" And they think of a bunch of types of disaster, decide that each wouldn't be bad enough, and end up with a low P(doom).

But the question should really be, "For each material condition that humans depend on to live, could runaway AI alter that condition enough to make human life nonviable?" So instead of listing out types of known disaster, we list out everything humans depend on — like land, clean water, breathable atmosphere, and so on. Then for each one, we ask, "Is there some way an unaligned optimizer running on our planet could take this thing away?"

Can a runaway AI take all the farmland away? Well, how could a particular piece of land be taken away? Buy it. Build a datacenter on it. Now that land can't be used for farming because there's a big hot datacenter on it. Do people buy up land and build datacenters on it today? Sure, and sometimes to the annoyance of the neighbors. What do you need to do that? Money. Can AI agents earn money from trade with humans, or with other AI agents? They sure can. Could AI agents be sufficiently economically dominant that they can literally outbid humanity for ownership of all the land? Hmm... that's not as easy as "could it cause a nuclear war big enough to kill everyone."

So the advice here could be summed up as something like: When you notice that you're brainstorming an exhaustive list of cases, first check that they're cases of the right nature.

I don't think this is a fallacy. If it was, one of the most powerful and common informal inference forms (IBE a.k.a. Inference to the Best Explanation / abduction) would be inadmissible. That would be absurd. Let me elaborate.

IBE works by listing all the potential explanations that come to mind, subjectively judging how good they are (with explanatory virtues like simplicity, fit, internal coherence, external coherence, unification, etc) and then inferring that the best explanation is probably correct. This involves the assumption that the probability is small that the true explanation is not among those which were considered. Sometimes this assumption seems unreasonable, in which case IBE shouldn't be applied. That's mostly the case if all considered explanations seem bad.

However, in many cases the "grain of truth" assumption (the true explanation is within the set of considered explanations) seems plausible. For example, I observe the door isn't locked. By far the best (least contrived) explanation I can think of seems to be that I forgot to lock it. But of course there is a near infinitude of explanations I didn't think of, so who is to say there isn't an unknown explanation which is even better than the one about my forgetfulness? Well, it just seems unlikely that there is such an explanation.

And IBE isn't just applicable to common everyday explanations. For example, the most common philosophical justification that the external world exists is an IBE. The best explanation for my experience of a table in front of me seems to be that there is a table in front of me. (Which interacts with light, which hits my eyes, which I probably also have, etc.)

Of course, in other cases, applications of IBE might be more controversial. However, in practice, if Alice makes an argument based on IBE, and Bob disagrees with its conclusion, this is commonly because Bob thinks Alice made a mistake when judging which of the explanations she considered is the best. In which case Bob can present reasons which suggest that, actually, explanation x is better than explanation y, contrary to what Alice assumed. Alice might be convinced by these reasons, or not, in which case she can provide the reasons why she still believes that y is better than x, and so on.

In short, in many or even most cases where someone disagrees with a particular application of IBE, their issue is not with IBE itself, but what the best explanation is. Which suggests the "grain of truth" assumption is often reasonable.

Most examples of bad reasoning, that are common amongst smart people, are almost good reasoning. Listing out all the ways something could happen is good, if and only if you actually list out all the ways something could happen

Well, that's clearly almost always impossible (there are almost infinitely many possible explanations for almost anything), so we can't make an exhaustive list. Moreover, "should" implies "can", so, by contraposition, if we can't list them, it's not the case that we should list them.

, or at least manage to grapple with most of the probability mass.

But that's backwards. IBE is a method which assigns probability to the best explanation based on how good it is (in terms of explanatory virtues) and based on being better than the other considered explanations. So IBE is a specific method for coming up with probabilities. It's not just stating your prior. You can't argue about purely subjective priors (that would be like arguing about taste) but you can make arguments about what makes some particular explanation good, or bad, or better than others. And if you happen to think that the "grain of truth" assumption is not plausible for a particular argument, you can also state that. (Though the fact that this is rather rarely done in practice suggests it's in general not such a bad assumption to make.)

I once quit a job in part^[1] because my manager insisted on calling one of these free associations 'rigorous and exhaustive'.

Exhaustive Free Association is a step in a chain of reasoning where the logic goes "It's not A, it's not B, it's not C, it's not D, and I can't think of any more things it could be!"^[1] Once you spot it, you notice it all the damn time.

This description skips over the fallacy part of the fallacy. On its own, the sentence in quotes sounds like a potentially productive contribution to a discussion.

Often related to maneuver warfare as well, ie making you expend more energy than you opponent does by putting the onus of refutation on you.

Some of this might be conflation between within-model predictions and overall predictions that account for model uncertaintly and unknown unknowns. Within-model predictions are in any case very useful as exercises for developing/understanding models, and as anchors for overall predictions. So it's good actually (rather than a problem) when within-model predictions are being made (based on whatever legible considerations that come to mind), including when they are prepared as part of the context before making an overall prediction, even for claims/predictions that are poorly understood and not properly captured by such models.

The issue is that when you run out of models and need to incorporate unknown unknowns, the last step that transitions from your collection of within-model prediction anchors to an overall prediction isn't going to be legible (otherwise it would just be following another model, and you'd still need to take that last step eventually). It's an error to give too much weight to within-model anchors (rather than some illegible prior) when the claim/prediction is overall poorly understood, but also sometimes the illegible overall assessment just happens to remain close to those anchors. And even base rates (reference classes) is just another model, it shouldn't claim to be the illegible prior at the end of this process, not when the claim/prediction remains poorly understood (and especially not when its understanding explicitly disagrees with the assumptions for base rate models).

So when you happen to disagree about the overall prediction, or about the extent to which the claim/prediction is well-understood, a prediction that happens to remain close to the legible anchors would look like it's committing the error described in the post, but it's not necessarily always (or often) the case. The only way to resolve such disagreements would be by figuring out how the last step was taken, but anything illegible takes a book to properly communicate. There's not going to be a good argument, for any issue that's genuinely poorly understood. The trick is usually to find related but different claims that can be understood better.

I personally just model errors like that as "projection". The error here is "I can't think of any more possibilities, therefore, more possibilities do not exist". It's very common for people to assume that other things are bounded by the same limitations as they are. The concept of "unknown unknowns" is related here as well.

More generally, when people talk about life and reality, they talk about themselves, even if they do not realize it. They assume their map is the territory. For instance, if a person says "Life is suffering", that may be true for them, and every counter-argument they hear may even evaluate to false in their model of reality, but that still doesn't mean it's true for everyone. 
Another comment mentioned "Proof by failure of imagination" and I like that name, since the fallacy is an error which occurs in a person. When we say "logical error" we don't mean that there's an error in logic itself, but in its use. If something is implicit for long enough, we risk forgetting it (I think this happened to morality. Now certain things are considered good in an absolute sense, rather than in a context)

If somebody uses a "proof by contradition", then a single example is enough (∃), but this argument is in the other direction, so one needs to show that something is true for all examples (∀) and not just some (∃). The only reason I can think of that somebody would make this error, is that they consider the examples they thought of to be "the best". If you can refute the best argument for why something would happen, it's easier to assume that it won't (I guess this is what steelmanning is?). This method works fine for smaller problem spaces, but quickly grows useless because of the inherent asymmetry between attacking and defending

Did you use EFA to conclude that EFA is the worst, common bad argument?