One of the most underrated essays in the post-Sequences era of Eliezer Yudkowsky's corpus is "Four Layers of Intellectual Conversation". The degree to which this piece of wisdom has fallen into tragic neglect in these dark ages of the 2020s may be related to its ephemeral form of publication: it was originally posted as a status update on Yudkowsky's Facebook account on 20 December 2016 and subsequently mirrored on Alyssa Vance's The Rationalist Conspiracy blog, which has since gone offline. (The first link in this paragraph is to an archive of the Rationalist Conspiracy post.)

In the post, Yudkowsky argues that a structure of intellectual value necessarily requires four layers of conversation: thesis, critique, response, and counter-response (which Yudkowsky indexes from zero as layers 0, 1, 2, and 3).

The importance of critique is already widespread common wisdom: if a thesis is advanced and promulgated without any serious effort to examine why it might be in error, then it likely is in error, both because it can't have incorporated corrections from critiques (which are ex hypothesi absent) and because the author lacks incentives to offer a correct thesis in the first place: if being right is difficult and there's no social penalty for being wrong, then most humans will inexorably find themselves on the easy course of being wrong even without any conscious intent to deceive. That is, in the words of the post, the problem with "a conversation consisting of people saying X and nobody saying 'hey maybe not-X'" is that "people could say stupid things about X, and nobody would call them on the stupidity." Yudkowsky aptly concludes: "Yikes!"

Yudkowsky's key observation going beyond common wisdom is that the necessity of social incentives to be correct also applies to the level-1 critique and level-2 response, not just the level-0 thesis—and moreover, that the higher levels are critical for the lower levels to maintain their force. The mere existence of level-1 critics won't suffice to keep level-0 thesis-proposers on their toes, if the level-1 critics are themselves not on their toes because they don't anticipate being held to account by level-2 responses. Likewise, level-2 responses won't suffice to keep level-1 critics on their toes if the level-2 responders don't anticipate being held to account by level-3 counter-responses. Without all four levels, the whole structure comes apart.

Yudkowsky offers public debates about evolution and molecular nanotechnology as examples of discourses with a missing level 3. If biologists explain evolution (level-0 thesis), religious scholars insist that God must have started it all (level-1 critique), biologists explain leading theories of abiogenesis (level-2 response), but religious scholars don't engage with the abiogenesis work, then the conversation has failed to secure a level-3 counter-response.

It matters that the higher levels are being held to a high enough standard that people would lose face if they played dumb. If K. Eric Drexler writes technical books and papers about the possibilities of nanotechnology (level-0 thesis), Richard Smalley objects that manipulator arms themselves made of atoms would be too "fat" and "sticky" to work as a molecular assembler and that this problem is fundamentally uncircumventable (level-1 critique), Drexler et al. reply that biological ribosomes demonstrate that the problem is not fundamentally uncircumventable even though Drexler's proposals have a "mechanical" rather than "biological" character (level-2 response), and Smalley objects that biological systems can't work with the materials used in technology and that Drexler has departed from real chemistry (level-3 counter-response), then all four levels are formally present, but one is left with disquieting sense that the level-3 counter-response has failed to truly connect with the level-2 response. (Drexler et al.'s level-2 response had brought up biology as an existence proof that the "fat finger" problem didn't sink the entire idea of nanotechnology; pointing out that biology can't do the things that Drexler had conjectured nanotechnology could, would seem to be missing the point.)

Yudkowsky laments that the academic journal system, with the possible exception of analytic philosophy, mostly only canonizes levels 0–2: it's uncommon to see a journal article that's a reply to a reply to a reply to another. To the extent that real intellectual progress is being made in most fields, the real work is probably happening at conferences or on email lists, with the journals merely recording the work after the fact. Yudkowsky sings the praises of transhumanist mailing lists of the late '90s, where people who might otherwise succumb to the temptation to play dumb were kept in check for fear of Robin Hanson's clinically precise rebuttals. Nick Bostrom's 2014 Superintelligence merely packaged up for the public the outcome of a hard-fought discourse that had occurred elsewhere.

A shortcoming of the original post is a lack of concrete examples (with labeled levels) of the four levels of conversation succeeding rather than failing. (We didn't get much detail about exactly what happened on that mailing list.)

The impact of the replication crisis on the study of priming effects might be a candidate. In 1996's "Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action", John A. Bargh and collaborators reported that college students directed to solve a puzzle involving words related to elderly people walked slower when leaving the lab (level-0 thesis). Sixteen years later, in "Behavioral Priming: It's All in the Mind, but Whose Mind?", Stéphane Doyen et al. ran a replication that failed to reproduce the original result on walking speed when the experimenter administering the puzzle was blinded to the hypothesis being tested, but did reproduce the result when the experimenter was led to believe that there would be a priming effect (level-1 critique). Bargh wrote a blog post, "Nothing in Their Heads", arguing that the experimenter was blinded in the original 1996 study, that Doyen et al. over-primed with too many elderliness-related words (which Bargh argued could destroy the effect), and that Doyen et al. didn't check if subjects had slowness-related stereotypes about the elderly (level-2 response). Though the original post's comment section seems to have been lost to history, science journalist Ed Yong documented responses to Bargh's post by commenters on the post and by coauthors of Doyen et al., claiming inaccuracies in the post, and that, in any case, a truly robust priming effect wouldn't be so fragile to such small changes in the study design (level-3 counter-response).

Nor did the conversation about this particular paper drop silently into the void: soon, the famed Daniel Kahneman would write a letter to priming research practitioners named to him by Bargh on bringing more rigorous study designs to the field, which has continued to be plagued by replication difficulties. The discussion made an impact on Society's collective beliefs. The attempt at discourse was more than a noble gesture. It hadn't all been for nothing.

A natural question to ask about the four-levels framework is: why four levels, specifically? Doesn't the recursion of level n needing level n + 1 go off to infinity?

The original post leaves the question unanswered, but a potential answer can be found in Yudkowsky's tongue-in-cheek Law of Ultrafinite Recursion, which states that, in practice, infinite recursions are at most three levels deep. The Law of Ultrafinite Recursion is deliberately silly if construed as a literal claim about computer science but is surprisingly fruitful as a claim about human psychology: it's pretty natural to ask what Alice thinks that Bob thinks about Carol, but asking what Alice thinks that Bob thinks that Carol thinks about Dave feels like a stretch.

If the limited human grasp of recursion rounds "four" up to "infinity", then the chain of thesis–critique–response–counter-response is enough to establish the expectation of unlimited-depth accountability and remove the incentive to bluff. A different species with greater working memory capacity, whose members could follow a backwards induction farther, might need more counter-counter-responses and counter-counter-counter-responses to experience the same salutary effect.

The four-levels model is about robust disagreements, which are usually pretty frustrating for all involved. No one likes being told they're wrong, especially by people who (so it always seems from the other side) are themselves obviously wrong.

The frustration is not optional. The recursive pressure forcing you to come up with your best arguments and responses to counter the adversary's critiques and counter-responses only works if the adversary is allowed to be frustrating; it's not their job to make it easy for you. Equivalently, it's not your job to make it easy for them. Only by facing this test can your combined efforts build an intellectual edifice guided by the beauty of your weapons.

Bargh's blog post complains that "oddly for an article that purported to fail to replicate one of [his] past studies", he wasn't asked to review Doyen et al. But it's not odd: journals generally want reviewers to be independent. For example, the International Committee of Medical Journal Editors recommends that peer reviewers should "declare their relationships and activities that might bias their evaluation of a manuscript and recuse themselves from the peer-review process if a conflict exists."

If Bargh were the one who got to decide who is allowed to speak on the record about potential flaws in Bargh et al. 1996, then Society would lose out on its chance to determine whether Bargh et al. 1996 is actually correct. Any single conversational locus that forgets or denies this obvious principle is at serious risk of degenerating into an echo chamber if it hasn't already.