tl;dr: rationalists concerned about AI risk often make claims that others consider not just unjustified, but unjustifiable using their current methodology, because of high-level disagreements about epistemology. If you actually want to productively discuss AI risk, make claims that can be engaged with by others who have a wide range of opinions about the appropriate level of Knightian uncertainty.

I think that many miscommunications about AI risk are caused by a difference between two types of norms for how to talk about the likelihoods of unprecedented events. I'll call these "inside view norms" versus "Knightian norms", and describe them as follows:

• Inside view norms: when talking to others, you report your beliefs directly, without adjusting for "Knightian uncertainty" (i.e. possible flaws or gaps in your model of the world that you can't account for directly).
• Knightian norms: you report beliefs adjusted for your best estimate of the Knightian uncertainty. For example, if you can't imagine any plausible future in which humanity and aliens end up cooperating with each other, but you think this is a domain which faces heavy Knightian uncertainty, then you might report your credence that we'll ever cooperate with aliens as 20%, or 30%, or 10%, but definitely nowhere near 0.

I'll give a brief justification of why Knightian norms seem reasonable to me, since I expect they're counterintuitive for most people on LW. On a principled level: when reasoning about complex domains like the future, the hardest part is often "knowing the right questions to ask", or narrowing down on useful categories at all. Some different ways in which a question might be the wrong one to ask:

• The question might have important ambiguities. For example, consider someone from 100 years ago asking "will humans be extinct in 1000 years?" Even for a concept like extinction that seems very black-and-white, there are many possible futures which are very non-central examples of either "extinct" or "not extinct" in the questioner's mind (e.g. all humans are digital; all humans are dramatically genetically engineered; all humans are merged with AIs; etc). And so it'd be appropriate to give an answer like "X% yes, Y% no, Z% this is the wrong question to ask".
• The question might be confused or ill-posed. For example, "how heavy is phlogiston?"
• You might be unable to conceptualize the actual answer. For example, suppose someone from 200 years ago asks "will physics be the fastest-moving science in the year 2023?" They think about all the sciences they know of, and all the possible future sciences they can imagine, and try to assign credences to them being the fastest-moving. But they'd very likely just totally fail to conceptualize the science that has turn out to be the fastest-moving: computer science (and machine learning more specifically). Even if they reason at a meta level "there are probably a bunch of future sciences I can basically not predict at all, so I should add credence to 'no'", the resulting uncertainty is Knightian in the sense that it's generated by reasoning about your ignorance rather than your actual models of the world.

I therefore consider Knightian norms to be appropriate when you're reasoning about a domain in which these considerations seem particularly salient. I give some more clarifications at the end of the post (in particular on why I think Knightian norms are importantly different from modesty norms). However, I'm less interested in debating the value of Knightian norms directly, and more interested in their implications for how to communicate. If one person is following inside view norms and another is following Knightian norms, that can cause serious miscommunications between them, especially if they disagree about which level of Knightian uncertainty is appropriate. So I endorse the following principle:

Communicate about your beliefs in ways that are robust to the level of Knightian uncertainty that your listeners interpret you as incorporating into your claims.

The justification for this principle is simple: we should prefer to focus discussions on the object-level models that we intended to communicate about, rather than on higher-level epistemological questions like "how much Knightian uncertainty is reasonable?"

Here's one hypothetical dialogue which fails to follow this principle:

Person 1: "My credence in AI x-risk is 90%"
Person 2: "But how can you have such high credence in such an unprecedented possibility, which it's very hard to gather direct empirical evidence about?"
Person 1: "I've thought about it a lot, and I see vanishingly few plausible paths to survival."
Person 2: "The idea that you can forecast these paths to the degree required to justify that level of confidence is incredibly implausible. Nobody can do that."
Person 1: [gets frustrated]
Person 2: [gets frustrated]

Some notes on this dialogue:

• Person 1 wanted to convey a case for why AI is an x-risk; they failed because their claims were inconsistent with the combination of a) person 2's belief that Knightian uncertainty in this domain is greater than 10%, and b) person 2's belief that person 1 was incorporating Knightian uncertainty into their credences.
• Person 2 wanted to learn about the case for AI x-risk; they also failed, because they focused on how person 1's credences seemed inconsistent with their beliefs about Knightian uncertainty, rather than digging into person 1's actual models.
• This was a dynamic which kept arising in my debate with Eliezer, and which made the debate much less productive than it would have been if we had focused on object-level issues, IMO.
• Almost everyone starts off with much less respect for Eliezer than I did, and therefore after hearing his claims doesn't even bother engaging, but rather jumps directly to a conclusion like "nobody can predict the future that well, therefore this person is talking nonsense" (where "nobody can do this" can be interpreted roughly as "the Knightian uncertainty in this domain is too high for those credences to be justifiable no matter how good their reasoning is").

By contrast, suppose that person 1 started by saying: "I've thought about the future of AI a lot, and I see vanishingly few plausible paths by which we survive." Person 2 might follow Knightian norms in which this epistemic state justifies a 10% credence in catastrophe, or a 50% credence, or a 90% credence. Importantly, though, that doesn't matter very much! In any of those cases they're able to directly engage with person 1's reasons for thinking there are few paths to survival, without getting sidetracked into the question of whether or why they follow Knightian norms to a different extent. (At some point the Knightian disagreement does become action-relevant - e.g. someone who believes in very strong Knightian uncertainty would want to focus much more on "preparing for the unexpected". But in practice I think it often has little or no action-relevance, because most of the time the best strategy is whichever one is best justified by your actual models.)

Am I just recommending "don't use credences directly in cases where others are likely following Knightian norms?" I do think that's a good first step, but I also think there's more to it than that. In particular, I advise:

• Where possible, make claims about your models rather than about reality directly (e.g. "I see vanishingly few plausible paths by which we survive" rather than "we're all going to die").
• When making claims about reality directly, where possible choose phrases which would still be accurate even if you were incorporating significantly more or less Knightian uncertainty (e.g. "we're probably all going to die" rather than "we're all going to die").
• When giving numerical credences, explicitly mention that these credences would be different if you had different views about Knightian uncertainty (e.g. "90% doom on my inside view, where that doesn't involve applying the level of Knightian uncertainty that most people apply in this domain").

I think relatively simple changes in accordance with these principles could significantly improve communication on these topics (with the main downside being a small loss of rhetorical force). So I strongly urge that people communicating outside the LW bubble (about AI x-risk in particular) follow these principles in order to actually communicate about the things they want to communicate about, rather than communicating implications like "I have a greater ability to predict the future than you thought humanly possible", which is what's currently happening.

Endnotes: Some further clarifications on why Knightian norms make sense:

1. Isn't it self-contradictory to try to estimate "unquantifiable" Knightian uncertainty? It is if you try to use the content of your model to make the estimate; but not if you use the type of evidence in favor of the model - how long you've thought about the topic, how many successful predictions you've made so far, etc. The stronger the types of evidence available, the less Knightian uncertainty you should estimate.
2. Don't Knightian norms lead people to report credences which don't sum to 1? Well, it depends. The best way to give explicit credences under Knightian norms probably looks something like "X% true, Y% false, Z% Knightian uncertainty". But in practice people (including myself) often round this off to "X% true, Y+Z% false", where the "true" side is the one that they consider to have the burden of proof. However, this is a pretty unprincipled approach which presumably gives rise to many inconsistencies. That's part of why I advocate for not reporting credences in these domains.
3. Knightian norms are closely related to modesty norms, but also importantly different. Knightian norms are just about your inability to dodge all the unknown unknowns; whether others agree or disagree is immaterial. In practice they're often invoked together, but for the sake of this post I'll treat them as fully separate.
4. An important aspect of Knightian uncertainty is that it's often not very decision-relevant, because it's hard for possibilities you can't conceptualize to inform your actual actions in the world. So you might wonder: why include it in your reports at all? My answer here is that it's often hard to tell where regular uncertainty ends and Knightian uncertainty begins, and so reporting your full uncertainty, including the Knightian component, is a good Schelling point.