Introduction

Edit 8/27: I think the tone of this post was not ideal, changing much of it! 

“After many years, I came to the conclusion that everything he says is false. . . . Every one of his arguments was tinged and coded with falseness and pretense. It was like playing chess with extra pieces. It was all fake.”

—Paul Postal (talking about Chomsky) (note, this is not exactly how I feel about Yudkowsky, I don’t think he’s knowingly dishonest, but I just thought it was a good quote and partially represents my attitude towards Yudkowsky).

Crosspost of this on my blog.  

In the days of my youth, about two years ago, I was a big fan of Eliezer Yudkowsky. I read his many, many writings religiously, and thought that he was right about most things. In my final year of high school debate, I read a case that relied crucially on the many worlds interpretation of quantum physics—and that was largely a consequence of reading through Eliezer’s quantum physics sequence. In fact, Eliezer’s memorable phrasing that the many worlds interpretation “wins outright given the current state of evidence,” was responsible for the title of my 44-part series arguing for utilitarianism titled “Utilitarianism Wins Outright.” If you read my early articles, you can find my occasional blathering about reductionism and other features that make it clear that my worldview was at least somewhat influenced by Eliezer.

But as I grew older and learned more, I came to conclude that much of what he said was deeply implausible.

Eliezer sounds good whenever he’s talking about a topic that I don’t know anything about. I know nothing about quantum physics, and he sounds persuasive when talking about quantum physics. But every single time he talks about a topic that I know anything about, with perhaps one or two exceptions, what he says is completely unreasonable, at least, when it’s not just advice about how to reason better. It is not just that I always end up disagreeing with him, it is that he says with almost total falsehood after falsehood, making it frequently clear he is out of his depth. And this happens almost every single time. It seems that, with few exceptions, whenever I know anything about a topic that he talks about, it becomes clear that his view is held confidently but very implausible. 

Why am I writing a hit piece on Yudkowsky? I certainly don’t hate him. In fact, I’d guess that I agree with him much more than almost all people on earth. Most people believe lots of outrageous falsehoods. And I think that he has probably done more good than harm for the world by sounding the alarm about AI, which is a genuine risk. And I quite enjoy his scrappy, willing-to-be-contrarian personality. So why him?

Part of this is caused by personal irritation. Each time I hear some rationalist blurt out “consciousness is just what an algorithm feels like from the inside,” I lose a year of my life and my blood pressure doubles (some have hypothesized that the explanation for the year of lost life involves the doubling of my blood pressure). And I spend much more time listening to Yukowsky’s followers say things that I think are false than most other people.

But a lot of it is that Yudkowsky has the ear of many influential people. He is one of the most influential AI ethicists around. Many people, my younger self included, have had their formative years hugely shaped by Yudkowsky’s views—on tons of topics. As Eliezer says:

In spite of how large my mistakes were, those two years of blog posting appeared to help a surprising number of people a surprising amount.

Quadratic Rationality expresses a common sentiment, that the sequences, written by Eliezer, have significantly shaped the world view of them and others. Eliezer is a hugely influential thinker, especially among effective altruists, who punch above their weight in terms of influence.

And Eliezer does often offer good advice. He is right that people often reason poorly, and there are ways people can improve their thinking. Humans are riddled by biases, and it’s worth reflecting on how that distorts our beliefs. I thus feel about him much like I do about Jordan Peterson—he provides helpful advice, but the more you listen, the more he sells you on a variety of deeply implausible, controversial views that have nothing to do with the self-help advice.

And the negative effects of Eliezer’s false beliefs have been significant. I’ve heard lots of people describe that they’re not vegan because of Eliezer’s animal consciousness views—views that are utterly nutty, as we’ll see. It is bad that many more people torture sentient beings on account of utterly loony beliefs about consciousness. Many people think that they won’t live to be 40 because they’re almost certain that AI will kill everyone, on account of Eliezer’s reasoning, and deference to Eliezer more broadly. Thinking that we all die soon can’t be good for mental health.

Eliezer’s influence is responsible for a narrow, insular way of speaking among effective altruists. It’s common to hear, at EA globals, peculiar LessWrong speak; something that is utterly antithetical to the goal of bringing new, normal non-nerds into the effective altruism movement. This is a point that I will assert without argument just based on my own sense of things—LessWrong speak masks confusion more than it enables understanding. People feel as though they’ve dissolved the hard problem by simply declaring that consciousness is what an algorithm feels like from the inside.

In addition, Eliezer’s views have undermined widespread trust in experts. They result in people thinking that they know better than David Chalmers about non-physicalism—that clever philosophers of mind are just morons who aren’t smart enough to understand Eliezer’s anti-zombie argument. Eliezer’s confident table pounding about quantum physics leads to people thinking that physicists are morons, incapable of understanding basic arguments. This undermining of trust in genuine authority results in lots of rationalists holding genuinely wacky views—if you think you are smarter than the experts, you are likely to believe crazy things.

Eliezer has swindled many of the smartest people into believing a whole host of wildly implausible things. Some of my favorite writers—e.g. Scott Alexander—seem to revere Eliezer. It’s about time someone pointed out his many false beliefs, the evaluation of which is outside of the normal competency of most people who do not know much about niche . If one of the world’s most influential thinkers is just demonstrably wrong about lots of topics, often in ways so egregious that they demonstrate very basic misunderstandings, then that’s quite newsworthy, just as it would be if a presidential candidate supported a slate of terrible policies.

The aim of this article is not to show that Eliezer is some idiot who is never right about anything. Instead, it is to show that Eliezer, on many topics, including ones where he describes agreeing with his position as being a litmus test for being sane, Eliezer is both immensely overconfident and demonstrably wrong. I think people, when they hear Eliezer express some view about some topic about which they’re unfamiliar, have roughly the following thought process:

Oh jeez, Eliezer thinks that most of the experts who think X are mistaken. I guess I should take seriously the hypothesis that X is wrong and that Eliezer has correctly identified an error in their reasoning. This is especially so given that he sounds convincing when he talks about X.

I think that instead they should have the following thought process:

I’m not an expert about X, but it seems like most of the experts about X think X or are unsure about it. The fact that Eliezer, who often veers sharply off-the-rails, thinks X gives me virtually no evidence about X. Eliezer, while being quite smart, is not rational enough to be worthy of significant deference on any subject, especially those subjects outside his area of expertise. Still though, he has some interesting things to say about AI and consequentialism that are sort of convincing. So it’s not like he’s wrong about everything or is a total crank. But he’s wrong enough, in sufficiently egregious ways, that I don’t really care what he thinks.

Eliezer is ridiculously overconfident and has a mediocre track record

Even the people who like Eliezer think that he’s wildly overconfident about lots of things. This is not without justification. Ben Garfinkel has a nice post on the EA forum running through Eliezer’s many, many mistaken beliefs that he held with very high confidence. Garfinkel suggests:

I think these examples suggest that (a) his track record is at best fairly mixed and (b) he has some tendency toward expressing dramatic views with excessive confidence.

Garfinkel runs through a series of incorrect predictions Eliezer has made. He predicted that nanotech would kill us all by 2010. Now, this was up until about 1999, when he was only about 20. So it’s not as probative as it would be if he made that prediction in 2005, for instance. But . . . still. If a guy has already incorrectly predicted that some technology would probably kill us soon, backed up by a rich array of arguments, and now he is predicting that some technology will kill us soon, backed up by a rich array of arguments, a reasonable inference is that, just like financial speculators who constantly predict recessions, this guy just has a bad habit of overpredicting doom.

I will not spend very much time talking about Eliezer’s views about AI, because they’re outside my area of expertise. But it’s worth noting that lots of people who know a lot about AI seem to think that Eliezer is ridiculously overconfident about AI. Jacob Cannell writes, in a detailed post arguing against Eliezer’s model:

My skill points instead have gone near exclusively towards extensive study of neuroscience, deep learning, and graphics/GPU programming. More than most, I actually have the depth and breadth of technical knowledge necessary to evaluate these claims in detail.

I have evaluated this model in detail and found it substantially incorrect and in fact brazenly naively overconfident.

. . .

Every one of his key assumptions is mostly wrong, as I and others predicted well in advance.

. . .

EY is just completely out of his depth here: he doesn't seem to understand how the Landauer limit actually works, doesn't seem to understand that synapses are analog MACs which minimally require OOMs more energy than simple binary switches, doesn't seem to have a good model of the interconnect requirements, etc.

I am also completely out of my depth here. Not only do I not understand how the Landauer limit works, I don’t even know what it is. But it’s worth noting that a guy who seems to know what he’s talking about thinks that many parts of Eliezer’s model are systematically overconfident, based on relatively egregious error.

Eliezer made many, many more incorrect predictions—let me just run through the list.

In 2001, and possibly later, Eliezer predicted that his team would build superintelligence probably between 2008-2010.

“In the first half of the 2000s, he produced a fair amount of technical and conceptual work related to this goal. It hasn't ultimately had much clear usefulness for AI development, and, partly on the basis, my impression is that it has not held up well - but that he was very confident in the value of this work at the time.”

Eliezer predicted that AI would quickly go from 0 to 100—that potentially over the course of a day, a single team would develop superintelligence. We don’t yet definitively know that that’s false but it almost certainly is.

There are other issues that are more debatable that Garfinkel highlights, that are probably instances of Eliezer’s errors. For most of those though, I don’t know enough to confidently evaluate them. But the worst part is that he has never acknowledged his mixed forecasting track record, and in fact, frequently acts as though he has a very good forecasting track record. This despite the fact that he often makes relatively nebulous predictions without giving credences, and then just gestures in the direction of having been mostly right about things when pressed about this. For example, he’ll claim that he came out better than Robin Hanson in the AI risk debate they had. Claiming that you were more right than someone, when you had wildly diverging models on a range of topics, is not a precise forecast (and in Eliezer’s case, is quite debatable). As Jotto999 notes:

In other domains, where we have more practice detecting punditry tactics, we would dismiss such an uninformative "track record".  We're used to hearing Tetlock talk about ambiguity in political statements.  We're used to hearing about a financial pundit like Jim Cramer underperforming the market.  But the domain is novel in AI timelines.

Even defenders of Eliezer agree that he’s wildly overconfident. Brian Tomasik, for example, says:

Really smart guy. His writings are “an acquired taste” as one of my friends put it, but I love his writing style, both for fiction and nonfiction. He’s one of the clearest and most enjoyable writers I’ve ever encountered.

My main high-level complaint is that Eliezer is overconfident about many of his beliefs and doesn’t give enough credence to other smart people. But as long as you take him with some salt, it’s fine.

Eliezer is in the top 10 list for people who have changed the way I see the universe.

Scott Alexander in a piece defending Eliezer says:

This is not to say that Eliezer – or anyone on Less Wrong – or anyone in the world – is never wrong or never overconfident. I happen to find Eliezer overconfident as heck a lot of the time.

The First Critical Error: Zombies

The zombie argument is an argument for non-physicalism. It’s hard to give a precise definition of non-physicalism, but the basic idea is that consciousness is non-physical in the sense that is it not reducible to the behavior of fundamental particles. Once you know the way atoms work, you can predict all the facts about chairs, tables, iron, sofas, and plants. Non-physicalists claim that consciousness is non-physical in the sense that it’s not explainable in that traditional way. The consciousness facts are fundamental—just as there are fundamental laws about the ways that particles behave, so too are there fundamental laws that govern that subjective experience arises in response to certain physical arrangements.

Let’s illustrate what a physicalist model of reality would work. Note, this is going to be a very simplistic and deeply implausible physicalist model; the idea is just to communicate the basic concept. Suppose that there are a bunch of blocks that move right every second. Assume these blocks are constantly conscious and consciously think “we want to move right.” A physicalist about this reality would think that to fully specify its goings-on, one would have to say the following:

Every second, every block moves right.

A non-physicalist in contrast might think one of the following two sets of rules specifies reality (the bolded thing is the name of the view):

Epiphenomenalism

Every second, every block moves right

Every second, every block thinks “I’d like to move right.”

Interactionism

Every second, every block thinks “I’d like to move right.”

Every time a block thinks “I’d like to move right,” it moves right.

The physical facts are facts about the way that matter behaves. Physicalists think once you’ve specified the way that matter behaves, that is sufficient to explain consciousness. Consciousness, just like tables and chairs, can be fully explained in terms of the behavior of physical things.

Non-physicalists think that the physicalists are wrong about this. Consciousness is its own separate thing that is not explainable just in terms of the way matter behaves. There are more niche views like idealism and panpsychism that we don’t need to go into, which say that consciousness is either fundamental to all particles or the only thing that exists, so let’s ignore them. The main view about consciousness is called dualism, according to which consciousness is non-physical and there are some psychophysical laws, that result in consciousness when there are particular physical arrangements.

There are broadly two kinds of dualism: epiphenomenalism and interactionism. Interactionism says that consciousness is causally efficacious, so the psychophysical laws describe that particular physical arrangements give rise to particular mental arrangements and also that those mental states cause other physical things. This can be seen in the block case—the psychophysical laws mean that the blocks give rise to particular conscious states that cause some physical things. Epiphenomenalism says the opposite—consciousness causes nothing. It’s an acausal epiphenomenon—the psychophysical laws go only one way. When there is a certain physical state, consciousness arises, but consciousness doesn’t cause anything further.

The zombie argument is an argument for non-physicalism about consciousness. It doesn’t argue for either an epiphenomenalist or interactionist account. Instead, it just argues against physicalism. The basic idea is as follows: imagine any physical arrangement that contains consciousness, for example, the actual world. Surely, we could imagine a world that is physically identical—where all the atoms, quarks, gluons, and such, move the same way—that doesn’t have consciousness. You could imagine an alternative version of me that is the same down to the atom.

Why think such beings are possible? They sure seem possible. I can quite vividly imagine a version of me that continues through its daily goings-on but that lacks consciousness. It’s very plausible that if something is impossible, there should be some reason that it is impossible—there shouldn’t just be brute impossibilities. The reason that married bachelors are impossible is that they require a contradiction—you can’t be both married and unmarried at the same time. But spelling out a contradiction in the zombie scenario has proved elusive.

I find the zombie argument quite convincing. But there are many smart people who disagree with it who are not off their rocker. Eliezer, however, has views on the zombie argument that demonstrate a basic misunderstanding of it—the type that would be cleared up in an elementary philosophy of mind class. In fact, Eliezer’s position on zombies is utterly bizarre; when describing the motivation for zombies, he writes what amounts to amusing fiction, trying to describe the motivation for zombies, but demonstrating that he has no idea what motivates belief in zombies. It would be like a Christian writer writing a thousand words eloquently steelmanning the problem of evil, but summarizing it as “atheists are angry at god because he creates things that they don’t like.”

What Eliezer thinks the zombie argument is (and what it is not)

Eliezer seems to think the zombie argument is roughly the following:

It seems like if you got rid of the world’s consciousness nothing would change because consciousness doesn’t do anything.

Therefore, consciousness doesn’t do anything.

Therefore it’s non-physical.

Eliezer then goes on an extended attack against premise 1. He argues that if it were true that consciousness does something, then you can’t just drain consciousness from the world and not change anything. So the argument for zombies hinges crucially on the assumption that consciousness doesn’t do anything. But he goes on to argue that consciousness does do something. If it didn’t do anything, what are the odds that when we talked about consciousness, our descriptions would match up with our conscious states? This would be a monumental coincidence, like it being the case that there are space aliens who work exactly the way you describe them to work, but your talk is causally unrelated to them—you’re just guessing and they happen to be exactly what you guess. It would be like saying “I believe there is a bridge in San Francisco with such and such dimensions, but the bridge existing has nothing to do with my talk about the bridge.” Eliezer says:

Your "zombie", in the philosophical usage of the term, is putatively a being that is exactly like you in every respect—identical behavior, identical speech, identical brain; every atom and quark in exactly the same position, moving according to the same causal laws of motion—except that your zombie is not conscious.

It is furthermore claimed that if zombies are "possible" (a term over which battles are still being fought), then, purely from our knowledge of this "possibility", we can deduce a priori that consciousness is extra-physical, in a sense to be described below; the standard term for this position is "epiphenomenalism".

(For those unfamiliar with zombies, I emphasize that this is not a strawman.  See, for example, the SEP entry on Zombies.  The "possibility" of zombies is accepted by a substantial fraction, possibly a majority, of academic philosophers of consciousness.)

Eliezer goes out of his way to emphasize that this is not a strawman. Unfortunately, it is a strawman. Not only that, Eliezer’s own source that he links to to describe how unstrawmanny it is shows that it is a strawman. Eliezer claims that the believers in zombies think consciousness is causally inefficacious and are called epiphenomenalists. But the SEP page he links to says:

True, the friends of zombies do not seem compelled to be epiphenomenalists or parallelists about the actual world. They may be interactionists, holding that our world is not physically closed, and that as a matter of actual fact nonphysical properties do have physical effects.

In fact, David Chalmers, perhaps the world’s leading philosopher of mind, says the same thing when leaving a comment below Eliezer’s post:

Someone e-mailed me a pointer to these discussions. I'm in the middle of four weeks on the road at conferences, so just a quick comment. It seems to me that although you present your arguments as arguments against the thesis (Z) that zombies are logically possible, they're really arguments against the thesis (E) that consciousness plays no causal role. Of course thesis E, epiphenomenalism, is a much easier target. This would be a legitimate strategy if thesis Z entails thesis E, as you appear to assume, but this is incorrect. I endorse Z, but I don't endorse E: see my discussion in "Consciousness and its Place in Nature", especially the discussion of interactionism (type-D dualism) and Russellian monism (type-F monism). I think that the correct conclusion of zombie-style arguments is the disjunction of the type-D, type-E, and type-F views, and I certainly don't favor the type-E view (epiphenomenalism) over the others. Unlike you, I don't think there are any watertight arguments against it, but if you're right that there are, then that just means that the conclusion of the argument should be narrowed to the other two views. Of course there's a lot more to be said about these issues, and the project of finding good arguments against Z is a worthwhile one, but I think that such an argument requires more than you've given us here.

The zombie argument is an argument for any kind of non-physicalism. Eliezer’s response is to argue that one particular kind of non-physicalism is false. That’s not an adequate response, or a response at all. If I argue “argument P means we have to accept views D, E, F, or I, and the response is ‘but view E has some problems’ that just means we should adopt views D, F, or I.”

But okay, what’s the error here? How does Eliezer’s version of the zombie argument differ from the real version? The crucial error is in his construction of premise 1. Eliezer assumes that, when talking about zombies, we are imagining just subtracting consciousness. He points out (rightly) that if consciousness is causally efficacious then if you only subtract consciousness, you wouldn’t have a physically identical world.

But the zombie argument isn’t about what would actually happen in our world if you just eliminated the consciousness. It’s about a physically identical world to ours lacking consciousness. Imagine you think that consciousness causes atoms 1, 2, and 3 to each move. Well then the zombie world would also involve them moving in the same physical way as they do when consciousness moves them. So it eliminates the experience, but it keeps a world that is physically identical.

This might sound pretty abstract. Let’s make it clearer. Imagine there’s a spirit called Casper. Casper does not have a physical body, does not emit light, and is physically undetectable. However, Casper does have conscious experience and has the ability to affect the world. Every thousand years, Casper can think “I really wish this planet would disappear,” and the planet would disappear. Crucially, we could imagine a world physically identical to the world with Casper, that just lacks Casper. This wouldn’t be what you would get if you just eliminated Casper—you’d also need to do something else to copy the physical effects that Casper has. So when writing the laws of nature for the world that copies Casper’s world, you’d also need to specify:

Oh, and also make one planet disappear every few months, specifically, the same ones Casper would have made disappear.

So the idea is that even if consciousness causes things, we could still imagine a physically identical world to the world where consciousness causes the things. Instead, the things would be caused the same physical way as they are with consciousness, but there would be no consciousness.

Thus, Eliezer’s argument fails completely. It is an argument against epiphenomenalism rather than an argument against zombieism. Eliezer thinks those are the same thing, but that is an error that no publishing academic philosopher could make. It’s really a basic error.

And when this is pointed out, Eliezer begins to squirm. For example, when responding to Chalmers’ comment, he says:

It seems to me that there is a direct, two-way logical entailment between "consciousness is epiphenomenal" and "zombies are logically possible".

If and only if consciousness is an effect that does not cause further third-party detectable effects, it is possible to describe a "zombie world" that is closed under the causes of third-party detectable effects, but lacks consciousness.

Type-D dualism, or interactionism, or what I've called "substance dualism", makes it impossible - by definition, though I hate to say it - that a zombie world can contain all the causes of a neuron's firing, but not contain consciousness.

You could, I suppose, separate causes into (arbitrary-seeming) classes of "physical causes" and "extraphysical causes", but then a world-description that contains only "physical causes" is incompletely specified, which generally is not what people mean by "ideally conceivable"; i.e., the zombies would be writing papers on consciousness for literally no reason, which sounds more like an incomplete imagination than a coherent state of affairs. If you want to give an experimental account of the observed motion of atoms, on Type-D dualism, you must account for all causes whether labeled "physical" or "extraphysical".

. . .

I understand that you have argued that epiphenomenalism is not equivalent to zombieism, enabling them to be argued separately; but I think this fails. Consciousness can be subtracted from the world without changing anything third-party-observable, if and only if consciousness doesn't cause any third-party-observable differences. Even if philosophers argue these ideas separately, that does not make them ideally separable; it represents (on my view) a failure to see logical implications.

Think back to the Casper example. Some physical effects in that universe are caused by physical things. Other effects in the universe are caused by nonphysical things (just one thing actually, Casper). This is not an arbitrary classification—if you believe that some things are physical and others are non-physical, then the division isn’t arbitrary. On type-D dualism, the consciousness causes things, and so the mirror world would just fill in the causal effects. A world description that contains only physical causes would be completely specified—it specifies all the behavior of the world, all the physical things, and just fails to specify the consciousness.

This is also just such cope! Eliezer spends an entire article saying, without argument, that zombieism = epiphenomenalism, assuming most people will believe him, and then when pressed on it, gives a barely coherent paragraph worth of justification for this false claim. It would be like it I argued against deontology by saying it was necessarily Kantian and arguing Kant was wrong, and then when called out on that by a leading non-Kantian deontologist, concocted some half-hearted justification for why they’re actually equivalent. That’s not being rational.

Even if we pretend, per impossible, that Eliezer’s extra paragraph refutes interactionist zombieism, it is not responsible to go through an entire article claiming that the only view that believes X is view Y, when that’s totally false, and then just later mention when pressed that there’s an argument for why believers in views other than X can’t believe Y.

In which Eliezer, after getting the basic philosophy of mind wrong, calls others stupid for believing in zombies

I think that the last section conclusively establishes that, at the very least, Eliezer’s views on the zombie argument both fail and evince a fundamental misunderstanding of the argument. But the most infuriating thing about this is Eliezer’s repeated insistence that disagreeing with him about zombies is indicative of fundamental stupidity. When explaining why he ignores philosophers because they don’t come to the right conclusions quickly enough, he says:

And if the debate about zombies is still considered open, then I'm sorry, but as Jeffreyssai says:  Too slow!  It would be one matter if I could just look up the standard answer and find that, lo and behold, it is correct.  But philosophy, which hasn't come to conclusions and moved on from cognitive reductions that I regard as relatively simple, doesn't seem very likely to build complex correct structures of conclusions.

Sorry - but philosophy, even the better grade of modern analytic philosophy, doesn't seem to end up commensurate with what I need, except by accident or by extraordinary competence.  Parfit comes to mind; and I haven't read much Dennett, but Dennett does seem to be trying to do the same sort of thing that I try to do; and of course there's Gary Drescher.  If there was a repository of philosophical work along those lines - not concerned with defending basic ideas like anti-zombieism, but with accepting those basic ideas and moving on to challenge more difficult quests of naturalism and cognitive reductionism - then that, I might well be interested in reading. 

(Eliezer wouldn’t like Parfit if he read more of him and realized he was a zombie-believing, non-physicalist, non-naturalist moral realist.)

There’s something infuriating about this. Making basic errors that show you don’t have the faintest grasp on what people are arguing about, and then acting like the people who take the time to get Ph.Ds and don’t end up agreeing with your half-baked arguments are just too stupid to be worth listening to is outrageous. And Eliezer repeatedly admonishes the alleged cognitive deficiency of us zombieists—for example:

I also want to emphasize that the “why so confident?” is a straw misquestion from people who can’t otherwise understand why I could be unconfident of many details yet still not take into account the conflicting opinion of people who eg endorse P-zombies.

It also seems to me that this is not all that inaccessible to a reasonable third party, though the sort of person who maintains some doubt about physicalism, or the sort of philosophers who think it’s still respectable academic debate rather than sheer foolishness to argue about the A-Theory vs. B-Theory of time, or the sort of person who can’t follow the argument for why all our remaining uncertainty should be within different many-worlds interpretations rather than slopping over outside, will not be able to access it.

We zombieists are apparently not reasonable third parties, because we can’t grasp Eliezer’s demonstrably fallacious reply to zombies. Being this confident and wrong is a significant mark against one’s reasoning abilities. If you believe something for terrible reasons, don’t update in response to criticisms over the course of decades, and then act like others who don’t agree with you are too stupid to get it, and in fact use that as one of your go-to examples of “things people stupider than I believe that I shouldn’t update on,” that seriously damages your credibility as a thinker. That evinces dramatic overconfidence, sloppiness, and arrogance.

The Second Critical Error: Decision Theory

Eliezer Yudkowsky has a decision theory called functional decision-theory. I will preface this by noting that I know much less about decision theory than I do about non-physicalism and zombies. Nevertheless, I know enough to get why Eliezer’s decision theory fails. In addition, most of this involves quoting people who are much more informed about decision theory than I am.

There are two dominant decision theories, both of which Eliezer rejects. The first is called causal decision theory. It says that when you have multiple actions that you can take, you should take the action that causes the best things. So, for example, if you have two actions, one of which would cause you to get 10 dollars, the other of which would cause you to get five dollars, and the final of which would cause you to get nothing, you should take the first action because it causes you to be richest at the end.

The next popular decision theory is called evidential decision theory. It says you should take the action where after you take that action you’ll expect to have the highest payouts. So in the earlier case, it would also suggest taking the first action because after you take that action, you’ll expect to be five dollars richer than if you take the second action, and ten dollars richer than if you take the third action.

These sound similar, so you might wonder where they come apart. Let me preface this by saying that I lean towards causal decision theory. Here are some cases where they give diverging suggestions:

Newcombe’s problem: there is a very good predictor who guessed whether you’d take two boxes or one box. If you take only one box, you’d take box A. If the guesser predicted that you’d take box A, they put a million dollars in box A. If they predicted you’d take both boxes, they put nothing into box A. In either case, they put a thousand dollars into box B.

Evidential decision theory would say that you should take only one box. Why? Those who take one box almost always get a million dollars, while those who take two boxes almost always get a thousand dollars. Causal decision theory would say you should take two boxes. On causal decision theory, it doesn’t matter whether people who make decisions like you usually end up worse off—what maters is that, no matter whether there is a million dollars in box A, two-boxing will cause you to have a free thousand dollars, and that is good! The causal decision theorist would note that if you had a benevolent friend who could peek into the boxes and then give you advice about what to do, they’d be guaranteed to suggest that you take both boxes. I used to have the intuition that you should one box, but when I considered this upcoming case, I abandoned that intuition.

Smoker’s lesion: suppose that smoking doesn’t actually cause averse health outcomes. However, smokers do have much higher rates of cancer than non-smokers. The reason for that is that many people have a lesion on their lung that both causes them to be much more likely to smoke and more likely to get cancer. So if you know that someone smokes, you should think it much more likely that they’ll get cancer even though smoking doesn’t cause cancer. Suppose that smoking is fun and doesn’t cause any harm. Evidential decision theory would say that you shouldn’t smoke because smoking gives you evidence that you’ll have a shorter life. You should, after smoking, expect your life to be shorter because it gives you evidence that you had a lesion on your lung. In contrast, causal decision theory would instruct you to smoke because it benefits you and doesn’t cause any harm.

Eliezer’s preferred view is called functional decision theory. Here’s my summary (phrased in a maximally Eliezer like way):

Your brain is a cognitive algorithm that outputs decisions in response to external data. Thus, when you take an action like

take one box

that entails that your mental algorithm outputs

take one box

in Newcombe’s problem. You should take actions such that the algorithm that outputs that decision generates higher expected utility than any other cognitive algorithm.

On Eliezer’s view, you should one box, but it’s fine to smoke because whether your brain outputs “smoke” doesn’t affect whether there is a lesion on your lung, so smoking. Or, as the impressively named Wolfgang Schwarz summarizes:

In FDT, the agent should not consider what would happen if she were to choose A or B. Instead, she ought to consider what would happen if the right choice according to FDT were A or B.

You should one box in this case because if FDT told agents to one box, they would get more utility on average than if FDT told agents to two box. Schwarz argues the first problem with the view is that it gives various totally insane recommendations. One example is a blackmail case. Suppose that a blackmailer will, every year, blackmail one person. There’s a 1 in a googol chance that he’ll blackmail someone who wouldn’t give in to the blackmail and a googol-1/googol chance that he’ll blackmail someone who would give in to the blackmail. He has blackmailed you. He threatens that if you don’t give him a dollar, he will share all of your most embarrassing secrets to everyone in the world. Should you give in?

FDT would say no. After all, agents who won’t give in are almost guaranteed to never be blackmailed. But this is totally crazy. You should give up one dollar to prevent all of your worst secrets from being spread to the world. As Schwarz says:

FDT says you should not pay because, if you were the kind of person who doesn't pay, you likely wouldn't have been blackmailed. How is that even relevant? You are being blackmailed. Not being blackmailed isn't on the table. It's not something you can choose.

Schwarz has another even more convincing counterexample:

Moreover, FDT does not in fact consider only consequences of the agent's own dispositions. The supposition that is used to evaluate acts is that FDT in general recommends that act, not just that the agent herself is disposed to choose the act. This leads to even stranger results.

Procreation. I wonder whether to procreate. I know for sure that doing so would make my life miserable. But I also have reason to believe that my father faced the exact same choice, and that he followed FDT. If FDT were to recommend not procreating, there's a significant probability that I wouldn't exist. I highly value existing (even miserably existing). So it would be better if FDT were to recommend procreating. So FDT says I should procreate. (Note that this (incrementally) confirms the hypothesis that my father used FDT in the same choice situation, for I know that he reached the decision to procreate.)

Schwarz’s entire piece is very worth reading. It exposes various parts of Soares and Yudkowsky’s paper that rest on demonstrable errors. Another good piece that takes down FDT is MacAskill’s post on LessWrong. He starts by laying out the following plausible principle:

Guaranteed Payoffs: In conditions of certainty — that is, when the decision-maker has no uncertainty about what state of nature she is in, and no uncertainty about the utility payoff of each action is — the decision-maker should choose the action that maximises utility. 

This is intuitively very obvious. If you know all the relevant facts about how the world is, and one act gives you more rewards than another act, you should take the first action. But MacAskill shows that FDT violates that constraint over and over again.

Bomb. 

You face two open boxes, Left and Right, and you must take one of them. In the Left box, there is a live bomb; taking this box will set off the bomb, setting you ablaze, and you certainly will burn slowly to death. The Right box is empty, but you have to pay $100 in order to be able to take it. 

A long-dead predictor predicted whether you would choose Left or Right, by running a simulation of you and seeing what that simulation did. If the predictor predicted that you would choose Right, then she put a bomb in Left. If the predictor predicted that you would choose Left, then she did not put a bomb in Left, and the box is empty. 

The predictor has a failure rate of only 1 in a trillion trillion. Helpfully, she left a note, explaining that she predicted that you would take Right, and therefore she put the bomb in Left. 

You are the only person left in the universe. You have a happy life, but you know that you will never meet another agent again, nor face another situation where any of your actions will have been predicted by another agent. What box should you choose?  

The right action, according to FDT, is to take Left, in the full knowledge that as a result you will slowly burn to death. Why? Because, using Y&S’s counterfactuals, if your algorithm were to output ‘Left’, then it would also have outputted ‘Left’ when the predictor made the simulation of you, and there would be no bomb in the box, and you could save yourself $100 by taking Left. In contrast, the right action on CDT or EDT is to take Right.

The recommendation is implausible enough. But if we stipulate that in this decision-situation the decision-maker is certain in the outcome that her actions would bring about, we see that FDT violates Guaranteed Payoffs. 

You can read MacAskill’s full post to find even more objections. He shows that Yudkowsky’s view is wildly indeterminate, incapable of telling you what to do, and also involves a broad kind of hypersensitivity, where however one defines “running the same algorithm” becomes hugely relevant, and determines very significant choices in seemingly arbitrary ways. The basic point is that Yudkowsky’s decision theory is totally bankrupt and implausible, in ways that are evident to those who know about decision theory. It is much worse than either evidential or causal decision theory.

The Third Critical Error: Animal Consciousness

(This was already covered here—if you’ve read that article skip this section and control F conclusion.)

Perhaps the most extreme example of an egregious error backed up by wild overconfidence occured in this Facebook debate about animal consciousness. Eliezer Yudkowsky expressed his view that pigs and almost all animals are almost certainly not conscious. Why is this? Well, as he says:

However, my theory of mind also says that the naive theory of mind is very wrong, and suggests that a pig does not have a more-simplified form of tangible experiences. My model says that certain types of reflectivity are critical to being something it is like something to be. The model of a pig as having pain that is like yours, but simpler, is wrong. The pig does have cognitive algorithms similar to the ones that impinge upon your own self-awareness as emotions, but without the reflective self-awareness that creates someone to listen to it.

Okay, so on this view, one needs to have reflective processes in order to be conscious. One’s brain has to model itself to be conscious. This doesn’t sound plausible to me, but perhaps if there’s overwhelming neuroscientific evidence, it’s worth accepting the view. And this view implies that pigs aren’t conscious, so Yudkowsky infers that they are not conscious.

This seems to me to be the wrong approach. It’s actually incredibly difficult to adjudicate between the different theories of consciousness. It makes sense to gather evidence for and against the consciousness of particular creatures, rather than starting with a general theory and using that to solve the problems. If your model says that pigs aren’t conscious, then that seems to be a problem with your model.

Mammals feel pain

I won’t go too in-depth here, but let’s just briefly review the evidence that mammals, at the very least, feel pain. This evidence is sufficiently strong that, as the SEP page on animal consciousness notes, “the position that all mammals are conscious is widely agreed upon among scientists who express views on the distribution of consciousness." The SEP page references two papers, one by Jaak Panksepp (awesome name!) and the other by Seth, Baars, and Edelman.

Let’s start with the Panksepp paper. They lay out the basic methodology, which involves looking at the parts of the brain that are necessary and sufficient for consciousness. So they see particular brain regions which are active during states when we’re conscious—and particularly correlate with particular mental states—and aren’t active when we’re not conscious. They then look at the brains of other mammals and notice that these features are ubiquitous in mammals, such that all mammals have the things that we know make us conscious in our brains. In addition, they act physically like we do when we’re in pain—they scream, they cry, their heart rate increases when they have a stressful stimulus, they make cost-benefit analyses where they’re willing to risk negative stimuli for greater reward. Sure looks like they’re conscious.

Specifically, they endorse a “psycho-neuro-ethological ‘‘triangulation’’ approach. The paper is filled with big phrases like that. What that means is that they look at various things that happen in the brain when we feel certain emotions. They observe that in humans, those emotions cause certain things—for example, being happy makes us more playful. They then look at mammal brains and see that they have the same basic brain structure, and this produces the same physical reactions—using the happiness example, this would also make the animals more playful. If they see that animals have the same basic neural structures as we do when we have certain experiences and that those are associated with the same physical states that occur when humans have those conscious states, they infer that the animals are having similar conscious states. If our brain looks like a duck’s brain when we have some experience, and we act like ducks do when they are in a comparable brain state, we should guess that ducks are having a similar experience. (I know we’re talking about mammals here, but I couldn’t resist the “looks like a duck, talks like a duck joke.”)

If a pig has a brain state that resembles ours when we are happy, tries to get things that make it happy, and produces the same neurological responses that we do when we’re happy, we should infer that pigs are not mindless automatons, but are, in fact, happy.

They then note that animals like drugs. Animals, like us, get addicted to opioids and have similar brain responses when they’re on opioids. As the authors note “Indeed, one can predict drugs that will be addictive in humans quite effectively from animal studies of desire.” If animals like the drugs that make us happy and react in similar ways to us, that gives us good reason to think that they are, in fact, happy.

They then note that the parts of the brain responsible for various human emotions are quite ancient—predating humans—and that mammals have them too. So, if the things that cause emotions are also present in animals, we should guess they’re conscious, especially when their behavior is perfectly consistent with being conscious. In fact, by running electricity through certain brain regions that animals share, we can induce conscious states in people—that shows that it is those brain states that are causing the various mental states.

The authors then run through various other mental states and show that those mental states are similar between humans and animals—animals have similar brain regions which provoke similar physical responses, and we know that in humans, those brain regions cause specific mental states.

Now, maybe there’s some magic of the human brain, such that in animal brains, the brain regions that cause qualia instead cause causally identical stuff but no consciousness. But there’s no good evidence for that, and plenty against. You should not posit special features of certain physical systems, for no reason.

Moving on to the Seth, Baars, and Edelman paper, they note that there are various features of consciousness, that differentiate conscious states from other things happening in the brain that don’t induce conscious states. They note:

Consciousness involves widespread, relatively fast, low-amplitude interactions in the thalamocortical core of the brain, driven by current tasks and conditions. Unconscious states are markedly different and much less responsive to sensory input or motor plans.

In other words, there are common patterns among conscious states. We can look at a human brain and see that the things that are associated with consciousness produce different neurological markers from the things that aren’t associated with consciousness. Features associated with consciousness include:

Irregular, low-amplitude brain activity: When we’re awake we have irregular low-amplitude brain activity. When we’re not conscious—e.g. in deep comas or anesthesia-induced unconsciousness—irregular, low-amplitude brain activity isn’t present. Mammal brains possess irregular, low-amplitude brain activity.

Involvement of the thalamocortical system: When you damage the thalamocortical system, that deletes part of one’s consciousness, unlike other systems. Mammals also have a thalamocortical system—just like us.

Widespread brain activity: Consciousness induces widespread brain activity. We don’t have that when things induce us not to be conscious, like being in a coma. Mammals do.

The authors note, from these three facts:

Together, these first three properties indicate that consciousness involves widespread, relatively fast, low-amplitude interactions in the thalamocortical core of the brain, driven by current tasks and conditions. Unconscious states are markedly different and much less responsive to sensory input or endogenous activity. These properties are directly testable and constitute necessary criteria for consciousness in humans. It is striking that these basic features are conserved among mammals, at least for sensory processes. The developed thalamocortical system that underlies human consciousness first arose with early mammals or mammal-like reptiles, more than 100 million years ago.

More evidence from neuroscience for animal consciousness:

Something else about metastability that I don’t really understand is also present in humans and animals.

Consciousness involves binding—bringing lots of different inputs together. In your consciousness, you can see the entire world at once, while thinking about things at the same time. Lots of different types of information are processed simultaneously, in the same way. Some explanations involving neural synchronicity have received some empirical support—and animals also have neural synchronicity, so they would also have the same kind of binding.

We attribute conscious experiences as happening to us. But mammals have a similar sense of self. Mammals, like us, process information relative to themselves—so they see a wall and process it relative to them in space.

Consciousness facilitates learning. Humans learn from conscious experiences. In contrast, we do not learn from things that do not impinge on our consciousness. If someone slaps me whenever I scratch my nose (someone does actually—crazy story), I learn not to scratch my nose. In contrast, if someone does a thing that I don’t consciously perceive when I scratch my nose, I won’t learn from it. But animals seem to learn to, and update in response to stimulus, just like humans do—but only when humans are exposed to things that affect their consciousness. In fact, even fish learn.

So there’s a veritable wealth of evidence that at least mammals are conscious. The evidence is less strong for organisms that are less intelligent and more distant from us evolutionarily, but it remains relatively strong for at least many fish. Overturning this abundance of evidence, that’s been enough to convince the substantial majority of consciousness researchers requires a lot of evidence. Does Yudkowsky have it?

Yudkowsky’s view is crazy, and is decisively refuted over and over again

No. No he does not. In fact, as far as I can tell, throughout the entire protracted Facebook exchange, he never adduced a single piece of evidence for his conclusion. The closest that he provides to an argument is the following:

I consider myself a specialist on reflectivity and on the dissolution of certain types of confusion. I have no compunction about disagreeing with other alleged specialists on authority; any reasonable disagreement on the details will be evaluated as an object-level argument. From my perspective, I’m not seeing any, “No, this is a non-mysterious theory of qualia that says pigs are sentient…” and a lot of “How do you know it doesn’t…?” to which the only answer I can give is, “I may not be certain, but I’m not going to update my remaining ignorance on your claim to be even more ignorant, because you haven’t yet named a new possibility I haven’t considered, nor pointed out what I consider to be a new problem with the best interim theory, so you’re not giving me a new reason to further spread probability density.”

What??? The suggestion seems to be that there is no other good theory of consciousness that implies that animals are conscious. To which I’d reply:

We don’t have any good theory about consciousness yet—the data is just too underdetermined. Just as you can know that apples fall when you drop them before you have a comprehensive theory of gravity, so too can you know some things about consciousness, even absent a comprehensive theory.

There are various theories that predict that animals are conscious. For example, integrated information theory, McFadden’s CEMI field theory, various Higher-Order theories, and the global workspace model will probably imply that animals are conscious. Eliezer has no argument to prefer his view to others.

Take the integrated information theory, for example. I don’t think it’s a great view. But at least it has something going for it. It has made a series of accurate predictions about the neural correlates of consciousness. Same with McFadden’s theory. It seems Yudkowsky’s theory has literally nothing going for it, beyond it sounding to Eliezer like a good solution. There is no empirical evidence for it, and, as we’ll see, it produces crazy, implausible implications. David Pearce has a nice comment about some of those implications:

Some errors are potentially ethically catastrophic. This is one of them. Many of our most intensely conscious experiences occur when meta-cognition or reflective self-awareness fails. Thus in orgasm, for instance, much of the neocortex effectively shuts down. Or compare a mounting sense of panic. As an intense feeling of panic becomes uncontrollable, are we to theorise that the experience somehow ceases to be unpleasant as the capacity for reflective self-awareness is lost? “Blind” panic induced by e.g. a sense of suffocation, or fleeing a fire in a crowded cinema (etc), is one of the most unpleasant experiences anyone can undergo, regardless or race or species. Also, compare microelectrode neural studies of awake subjects probing different brain regions; stimulating various regions of the “primitive” limbic system elicits the most intense experiences. And compare dreams – not least, nightmares – many of which are emotionally intense and characterised precisely by the lack of reflectivity or critical meta-cognitive capacity that we enjoy in waking life.

Yudkowsky’s theory of consciousness would predict that during especially intense experiences, where we’re not reflecting, we’re either not conscious or less conscious. So when people orgasm, they’re not conscious. That’s very implausible. Or, when a person is in unbelievable panic, on this view, they become non-conscious or less conscious. Pearce further notes:

Children with autism have profound deficits of self-modelling as well as social cognition compared to neurotypical folk. So are profoundly autistic humans less intensely conscious than hyper-social people? In extreme cases, do the severely autistic lack consciousness’ altogether, as Eliezer’s conjecture would suggest? Perhaps compare the accumulating evidence for Henry Markram’s “Intense World” theory of autism.

Francisco Boni Neto furthers:

many of our most intensely conscious experiences occur when meta-cognition or reflective self-awareness fails. Super vivid, hyper conscious experiences, phenomenic rich and deep experiences like lucid dreaming and ‘out-of-body’ experiences happens when higher structures responsible for top-bottom processing are suppressed. They lack a realistic conviction, specially when you wake up, but they do feel intense and raw along the pain-pleasure axis.

Eliezer just bites the bullet:

I’m not totally sure people in sufficiently unreflective flow-like states are conscious, and I give serious consideration to the proposition that I am reflective enough for consciousness only during the moments I happen to wonder whether I am conscious. This is not where most of my probability mass lies, but it’s on the table.

So when confronted with tons of neurological evidence that shutting down higher processing results in more intense conscious experiences, Eliezer just says that when we think that we have more intense experiences, we’re actually zombies or something? That’s totally crazy. It’s sufficiently crazy that I think I might be misunderstanding him. When you find out that your view says that people are barely conscious or non-conscious when they orgasm or that some very autistic people aren’t conscious, it makes sense to give up the damn theory!

And this isn’t the only bullet Eliezer bites. He admits, “It would not surprise me very much to learn that average children develop inner listeners at age six.” I have memories from before age 6—these memories would have to have been before I was conscious, on this view.

Rob Wiblin makes a good point:

[Eliezer], it’s possible that what you are referring to as an ‘inner listener’ is necessary for subjective experience, and that this happened to be added by evolution just before the human line. It’s also possible that consciousness is primitive and everything is conscious to some extent. But why have the prior that almost all non-human animals are not conscious and lack those parts until someone brings you evidence to the contrary (i.e. “What I need to hear to be persuaded is,”)? That just cannot be rational.

You should simply say that you are a) uncertain what causes consciousness, because really nobody knows yet, and b) you don’t know if e.g. pigs have the things that are proposed as being necessary for consciousness, because you haven’t really looked into it.

I agree with Rob. We should be pretty uncertain. My credences are maybe the following:

92% that at least almost all mammals are conscious.

80% that almost all reptiles are conscious.

60% that fish are mostly conscious.

30% that insects are conscious.

It’s about as likely that reptiles aren’t conscious as insects are. Because consciousness is private—you only know your own—we shouldn’t be very confident about any features of consciousness.

Based on these considerations, I conclude that Eliezer’s view is legitimately crazy. There is, quite literally, no good reason to believe it, and lots of evidence against it. Eliezer just dismisses that evidence, for no good reason, bites a million bullets, and acts like that’s the obvious solution.

Absurd overconfidence

The thing that was most infuriating about this exchange was Eliezer’s insistence that those who disagreed with him were stupid, combined with his demonstration that he had no idea what he was talking about. Condescension and error make an unfortunate combination. He says of the position that pigs, for instance, aren’t conscious:

It also seems to me that this is not all that inaccessible to a reasonable third party, though the sort of person who maintains some doubt about physicalism, or the sort of philosophers who think it’s still respectable academic debate rather than sheer foolishness to argue about the A-Theory vs. B-Theory of time, or the sort of person who can’t follow the argument for why all our remaining uncertainty should be within different many-worlds interpretations rather than slopping over outside, will not be able to access it.

Count me in as a person who can’t follow any arguments about quantum physics, much less the arguments for why we should be almost certain of many worlds. But seriously, physicalism? We should have no doubt about physicalism? As I’ve argued before, the case against physicalism is formidable. Eliezer thinks it’s an open-and-shut case, but that’s because he is demonstrably mistaken about the zombie argument against physicalism and the implications of non-physicalism. 

And that’s not the only thing Eliezer expresses insane overconfidence about. In response to his position that most animals other than humans aren’t conscious, David Pearce points out that you shouldn’t be very confident in positions that almost all experts disagree with you about, especially when you have a strong personal interest in their view being false. Eliezer replies:

What do they think they know and how do they think they know it? If they’re saying “Here is how we think an inner listener functions, here is how we identified the associated brain functions, and here is how we found it in animals and that showed that it carries out the same functions” I would be quite impressed. What I expect to see is, “We found this area lights up when humans are sad. Look, pigs have it too.” Emotions are just plain simpler than inner listeners. I’d expect to see analogous brain areas in birds.

When I read this, I almost fell out of my chair. Eliezer admits that he has not so much as read the arguments people give for widespread animal consciousness. He is basing his view on a guess of what they say, combined with an implausible physical theory for which he has no evidence. This would be like coming to the conclusion that the earth is 6,000 years old, despite near-ubiquitous expert disagreement, providing no evidence for the view, and then admitting that you haven’t even read the arguments that experts give in the field against your position. This is the gravest of epistemic sins.

Conclusion

This has not been anywhere near exhaustive. I haven’t even started talking about Eliezer’s very implausible views about morality (though I might write about that too—stay tuned), reductionism, modality, or many other topics. Eliezer usually has a lot to say about topics, and it often takes many thousands of words to refute what he’s saying.

I hope this article has shown that Eliezer frequently expresses near certainty on topics that he has a basic ignorance about, an ignorance so profound that he should suspend judgment. Then, infuriatingly, he acts like those who disagree with his errors are morons. He acts like he is a better decision theorist than the professional decision theorists, a better physicist than the physicists, a better animal consciousness researcher than the animal consciousness researchers, and a much better philosopher of mind than the leading philosophers of mind.

My goal in this is not to cause people to stop reading Eliezer. It’s instead to encourage people to refrain from forming views on things he says just from reading him. It’s to encourage people to take his views with many grains of salt. If you’re reading something by Eliezer and it seems too obvious, on a controversial issue, there’s a decent chance you are being duped.

I feel like there are two types of thinkers, the first we might call innovators and the second systematizers. Innovators are the kinds of people who think of wacky, out-of-the-box ideas, but are less likely to be right. They enrich the state of discourse by being clever, creative, and coming up with new ideas, rather than being right about everything. A paradigm example is Robin Hanson—no one feels comfortable just deferring to Robin Hanson across the board, but Robin Hanson has some of the most ingenious ideas.

Systematizers, in contrast, are the kinds of people who reliably generate true beliefs on lots of topics. A good example is Scott Alexander. I didn’t research Ivermectin, but I feel confident that Scott’s post on Ivermectin is at least mostly right.

I think people think of Eliezer as a systematizer. And this is a mistake, because he just makes too many errors. He’s too confident about things he’s totally ignorant about. But he’s still a great innovator. He has lots of interesting, clever ideas that are worth hearing out. In general, however, the fact that Eliezer believes something is not especially probative. Eliezer’s skill lies in good writing and ingenious argumentation, not forming true beliefs.