Background: As can be seen from some of the comments on this post, many people in the LessWrong community take an extreme stance on lying. A few days before I posted this, I was at a meetup where we played the game Resistance, and one guy announced before the game began that he had a policy of never lying even when playing games like that. It's such members of the LessWrong community that this post was written for. I'm not trying to encourage basically honest people with the normal view of white lies that they need to give up being basically honest.

Mr. Potter, you sometimes make a game of lying with truths, playing with words to conceal your meanings in plain sight. I, too, have been known to find that amusing. But if I so much as tell you what I hope we shall do this day, Mr. Potter, you will lie about it. You will lie straight out, without hesitation, without wordplay or hints, to anyone who asks about it, be they foe or closest friend. You will lie to Malfoy, to Granger, and to McGonagall. You will speak, always and without hesitation, in exactly the fashion you would speak if you knew nothing, with no concern for your honor. That also is how it must be.

- Rational!Quirrell, Harry Potter and the Methods of Rationality

This post isn't about HMPOR, so I won't comment on the fictional situation the quote comes from. But in many real-world situations, it's excellent advice.

If you're a gay teenager with homophobic parents, and there's a real chance they'd throw you out on the street if they found out you were gay, you should probably lie to them about it. Even in college, if you're still financially dependent on them, I think it's okay to lie. The minute you're no longer financially dependent on them, you should absolutely come out for your sake and the sake of the world. But it's OK to lie if you need to to keep your education on-track.

Oh, maybe you could get away with just shutting up and hoping the topic doesn't come up. When asked about dating, you could try to evade while being technically truthful: "There just aren't any girls at my school I really like." "What about _____? Why don't you ask her out?" "We're just friends." That might work. But when asked directly "are you gay?" and the wrong answer could seriously screw-up your life, I wouldn't bet too much on your ability to "lie with truths," as Quirrell would say.

I start with this example because the discussions I've seen on the ethics of lying on LessWrong (and everywhere, actually) tend to focus on the extreme cases: the now-cliché "Nazis at the door" example, or even discussion of whether you'd lie with the world at stake. The "teen with homophobic parents" case, on the other hand, might have actually happened to someone you know. But even this case is extreme compared to most of the lies people tell on a regular basis.

Widely-cited statistics claim that the average person lies once per day. I recently saw a new study (that I can't find at the moment) that disputed this, and claimed most people lie rather less often than that, but it still found most people lie fairly often. These lies are mostly "white lies" to, say, spare others' feelings. Most people have no qualms about those kind of lies. So why do discussions of the ethics of lying so often focus on the extreme cases, as if those were the only ones where lying is maybe possibly morally permissible?

At LessWrong there've been discussions of several different views all described as "radical honesty." No one I know of, though, has advocated Radical Honesty as defined by psychotherapist Brad Blanton, which (among other things) demands that people share every negative thought they have about other people. (If you haven't, I recommend reading A. J. Jacobs on Blanton's movement.) While I'm glad no one here is thinks Blanton's version of radical honesty is a good idea, a strict no-lies policy can sometimes have effects that are just as disastrous.

A few years ago, for example, when I went to see the play my girlfriend had done stage crew for, and she asked what I thought of it. She wasn't satisfied with my initial noncommittal answers, so she pressed for more. Not in a "trying to start a fight" way; I just wasn't doing a good job of being evasive. I eventually gave in and explained why I thought the acting had sucked, which did not make her happy. I think incidents like that must have contributed to our breaking up shortly thereafter. The breakup was a good thing for other reasons, but I still regret not lying to her about what I thought of the play.

Yes, there are probably things I could've said in that situation that would have been not-lies and also would have avoided upsetting her. Sam Harris, in his book Lying, spends a lot of arguing against lying in that way: he takes situations where most people would be tempted to tell a white lie, and suggesting ways around it. But for that to work, you need to be good at striking the delicate balance between saying too little and saying too much, and framing hard truths diplomatically. Are people who lie because they lack that skill really less moral than people who are able to avoid lying because they have it?

Notice the signaling issue here: Sam Harris' book is a subtle brag that he has the skills to tell people the truth without too much backlash. This is especially true when Harris gives examples from his own life, like the time he told a friend "No one would ever call you 'fat,' but I think you could probably lose twenty-five pounds." and his friend went and did it rather than getting angry. Conspicuous honesty also overlaps with conspicuous outrage, the signaling move that announces (as Steven Pinker put it) "I'm so talented, wealthy, popular, or well-connected that I can afford to offend you."

If you're highly averse to lying, I'm not going to spend a lot of time trying to convince you to tell white lies more often. But I will implore you to do one thing: accept other people's right to lie to you. About some topics, anyway. Accept that some things are none of your business, and sometimes that includes the fact that there's something which is none of your business.

Or: suppose you ask someone for something, they say "no," and you suspect their reason for saying "no" is a lie. When that happens, don't get mad or press them for the real reason. Among other things, they may be operating on the assumptions of guess culture, where your request means you strongly expected a "yes" and you might not think their real reason for saying "no" was good enough. Maybe you know you'd take an honest refusal well (even if it's "I don't want to and don't think I owe you that"), but they don't necessarily know that. And maybe you think you'd take an honest refusal well, but what if you're lying to yourself?

If it helps to be more concrete: Some men will react badly to being turned down for a date. Some women too, but probably more men, so I'll make this gendered. And also because dealing with someone who won't take "no" for an answer is a scarier experience with the asker is a man and the person saying "no" is a woman. So I sympathize with women who give made-up reasons for saying "no" to dates, to make saying "no" easier.

Is it always the wisest decision? Probably not. But sometimes, I suspect, it is. And I'd advise men to accept that women doing that is OK. Not only that, I wouldn't want to be part of a community with lots of men who didn't get things like that. That's the kind of thing I have in mind when I say to respect other people's right to lie to you.

All this needs the disclaimer that some domains should be lie-free zones. I value the truth and despise those who would corrupt intellectual discourse with lies. Or, as Eliezer once put it:

We believe that scientists should always tell the whole truth about science. It's one thing to lie in everyday life, lie to your boss, lie to the police, lie to your lover; but whoever lies in a journal article is guilty of utter heresy and will be excommunicated.

I worry this post will be dismissed as trivial. I simultaneously worry that, even with the above disclaimer, someone is going to respond, "Chris admits to thinking lying is often okay, now we can't trust anything he says!" If you're thinking of saying that, that's your problem, not mine. Most people will lie to you occasionally, and if you get upset about it you're setting yourself up for a lot of unhappiness. And refusing to trust someone who lies sometimes isn't actually very rational; all but the most prolific liars don't lie anything like half the time, so what they say is still significant evidence, most of the time. (Maybe such declarations-of-refusal-to-trust shouldn't be taken as arguments so much as threats meant to coerce more honesty than most people feel bound to give.)

On the other hand, if we ever meet in person, I hope you realize I might lie to you. Failure to realize a statement could be a white lie can create some terribly awkward situations.

Edits: Changed title, added background, clarified the section on accepting other people's right to lie to you (partly cutting and pasting from this comment).

Edit round 2: Added link to paper supporting claim that the average person lies once per day.

Quite a few people complain about the atheist/skeptic/rationalist communities being self-congratulatory. I used to dismiss this as a sign of people's unwillingness to admit that rejecting religion, or astrology, or whatever, was any more rational than accepting those things. Lately, though, I've started to worry.

Frankly, there seem to be a lot of people in the LessWrong community who imagine themselves to be, not just more rational than average, but paragons of rationality who other people should accept as such. I've encountered people talking as if it's ridiculous to suggest they might sometimes respond badly to being told the truth about certain subjects. I've encountered people asserting the rational superiority of themselves and others in the community for flimsy reasons, or no reason at all.

Yet the readiness of members of the LessWrong community to disagree with and criticize each other suggests we don't actually think all that highly of each other's rationality. The fact that members of the LessWrong community tend to be smart is no guarantee that they will be rational. And we have much reason to fear "rationality" degenerating into signaling games.

Previously: Mainstream Nutrition Science on Obesity, Atkins Redux, Did the US Government Give Us Absurd Advice About Sugar?

In this post, I'm going to deal with an issue that's central to Gary Taubes' critique of mainstream nutrition science: what causes obesity?

This is a post a post I found exceptionally difficult to write. You see, while his 2002 New York Times article portrays mainstream nutrition science as promoting a simplistic mirror-image of the Atkins diet, his books do manage to talk about the mainstream view that if you consume more calories than you burn you'll gain weight... sort of. As I looked closely at the relevant chapters of those books, it became less and less clear what view he's attributing to mainstream experts, or what his alternative is supposed to be.

Because this discussion may get confusing, I want to start by repeating what I said in my first post: the mainstream view is that people gain weight when they consume more calories than they burn, but both calorie intake and calorie expenditure are regulated by complicated mechanisms we don't fully understand yet.

Yet Taubes goes on at great length about how obesity has other causes beyond simple calorie math as if this were somehow a refutation of mainstream nutrition science. So I'm going to provide a series of quotes from relevant sources to show that the experts are perfectly aware of that fact. All of the following sources are ones Taubes cites as examples of how absurd the views of mainstream nutrition experts supposedly are:

Previously: Mainstream Nutrition Science on Obesity

Edit: In retrospect, I think it maybe should have combined this post with part 3. Unfortunately, the problem of what to do with existing comments makes that hard to fix now.

Taubes first made a name for himself as a low-carb advocate in 2002 with a New York Times article titled "What if It's All Been a Big Fat Lie?" When I first read this article, I was getting extremely suspicious by the second paragraph (emphasis added):

If the members of the American medical establishment were to have a collective find-yourself-standing-naked-in-Times-Square-type nightmare, this might be it. They spend 30 years ridiculing Robert Atkins, author of the phenomenally-best-selling ''Dr. Atkins' Diet Revolution'' and ''Dr. Atkins' New Diet Revolution,'' accusing the Manhattan doctor of quackery and fraud, only to discover that the unrepentant Atkins was right all along. Or maybe it's this: they find that their very own dietary recommendations -- eat less fat and more carbohydrates -- are the cause of the rampaging epidemic of obesity in America. Or, just possibly this: they find out both of the above are true.

When Atkins first published his ''Diet Revolution'' in 1972, Americans were just coming to terms with the proposition that fat -- particularly the saturated fat of meat and dairy products -- was the primary nutritional evil in the American diet. Atkins managed to sell millions of copies of a book promising that we would lose weight eating steak, eggs and butter to our heart's desire, because it was the carbohydrates, the pasta, rice, bagels and sugar, that caused obesity and even heart disease. Fat, he said, was harmless.

Atkins allowed his readers to eat ''truly luxurious foods without limit,'' as he put it, ''lobster with butter sauce, steak with béarnaise sauce . . . bacon cheeseburgers,'' but allowed no starches or refined carbohydrates, which means no sugars or anything made from flour. Atkins banned even fruit juices, and permitted only a modicum of vegetables, although the latter were negotiable as the diet progressed.

It's one thing to claim that, all else equal, low-carb diets have advantages over low-fat diets. It's another thing to claim you can eat unlimited amounts of fatty foods without gaining weight.

Related: Trusting Expert Consensus

Lately, I've been thinking a lot about whether we can find any clear exceptions to the general "trust the experts (when they agree)" heuristic. One example that keeps coming up—at least on LessWrong and related blogs—is Gary Taubes' claims about mainstream nutrition experts allegedly getting obesity horribly wrong.

Taubes is probably best-known for his book Good Calories, Bad Calories. I'd previously had a mildly negative impression of him from discussion of him on Yvain's old blog, particularly some of other posts Yvain and other people linked from there, such as this discussion of Taubes' "carbohydrate hypothesis" and especially this discussion of Taubes' attempt to refute the standard calories-in/calories-out model of weight.

But I figured maybe the criticism of Taubes I'd read hadn't been fair to him, so I decided to read him for myself... and holy crap, Taubes turned out to be far worse than I expected. I decided to write a post explaining why, and then realized that, even if I were somewhat selective about the issues I focused on, I had enough material for a whole series of posts, which I'll be posting over the course of the next week.

The problem with Taubes is not that everything he says is wrong. Much of it is ludicrously wrong, but that's only one half of the problem. The other half is that he says a fair number of things mainstream nutrition science would agree with, but then hides this fact, and instead pretends those things are a refutation of mainstream nutrition science. So it's worth starting with a brief in-a-nutshell version of what mainstream nutrition science actually says about obesity.

(The following summary is drawn from a number of sources, including this, this, and this. Everything I'm about to say will be discussed in much greater detail in subsequent posts.)

Here it goes: people gain weight when they consume more calories than they burn. But both calorie intake and calorie expenditure are regulated by complicated mechanisms we don't fully understand yet. This means the causes of overweight and obesity* are also complicated and not fully understood. It is, however, worth watching out for foods with lots of added fat and sugar, if only because they're an easy way to consume way too many calories.

We currently don't have any great solutions to the problem of overweight and obesity. If you consume fewer calories than you burn, you will lose weight, but sticking to a diet is hard. It's relatively easy to lose weight in the short run, and it's possible to do so on a wide variety of diets, but only a small percentage of people keep the weight off over the long run.

As for low-carb diets, people do lose weight on them, but they do so because low-carb diets generally lead people to restrict their calorie intake even when they aren't actively counting calories. For one thing, it's hard to consume as many calories when you drastically restrict the range of foods you can eat. There's also some evidence that low-carb diets may have some advantages. in terms of, say, warding off hunger, but the evidence is mixed. There's certainly no basis for claiming low-carb diets as a magic bullet for the problems of overweight and obesity.

The above points are not the only issues at stake in Taubes' writings on nutrition. Admittedly, he covers a huge amount of ground, from the relationship between sugar and diabetes to the relationship between fat intake and heart disease to the alleged dangers of extremely-low carbohydrate diets. However, I'll be focusing on his claims about the causes of and solutions to the problems of overweight and obesity, because that seems to be the main thing people talk about when they talk about Taubes supposedly showing how wrong mainstream experts can be.

I'll also focus heavily on how Taubes misrepresents the views of mainstream experts on obesity. In the next post, though, I'll be temporarily setting that issue aside in order to look at what Taubes is proposing as an alternative. This will involve examining some claims made by Dr. Robert Atkins, whose ideas' Taubes champions.

*Note: if the use of "overweight" as a noun sounds weird to you, it does to me too, but I discovered as I researched this article that it's standard usage in the literature on the subject. I came to realize there's a good reason for this usage: it's inaccurate to talk about the problem solely in terms of "obesity," but constantly saying "the problem of people being overweight and obese" gets really wordy.

Next: Atkins Redux

Consider the following commonly-made argument: cryonics is unlikely to work. Trained rationalists are signed up for cryonics at rates much greater than the general population. Therefore, rationalists must be pretty gullible people, and their claims to be good at evaluating evidence must be exaggerations at best.

This argument is wrong, and we can prove it using data from the last two Less Wrong surveys.

The question at hand is whether rationalist training - represented here by extensive familiarity with Less Wrong material - makes people more likely to believe in cryonics.

We investigate with a cross-sectional study, looking at proto-rationalists versus experienced rationalists. Define proto-rationalists as those respondents to the Less Wrong survey who indicate they have been in the community for less than six months and have zero karma (usually indicative of never having posted a comment). And define experienced rationalists as those respondents to the Less Wrong survey who indicate they have been in the community for over two years and have >1000 karma (usually indicative of having written many well-received posts).

By these definitions, there are 93 proto-rationalists, who have been in the community an average of 1.3 months, and 134 experienced rationalists, who have been in the community an average of 4.5 years. Proto-rationalists generally have not read any rationality training material - only 20/93 had read even one-quarter of the Less Wrong Sequences. Experienced rationalists are, well, more experienced: two-thirds of them have read pretty much all the Sequence material.

Proto-rationalists thought that, on average, there was a 21% chance of an average cryonically frozen person being revived in the future. Experienced rationalists thought that, on average, there was a 15% chance of same. The difference was marginally significant (p < 0.1).

Marginal significance is a copout, but this isn't our only data source. Last year, using the same definitions, proto-rationalists assigned a 15% probability to cryonics working, and experienced rationalists assigned a 12% chance. We see the same pattern.

So experienced rationalists are consistently less likely to believe in cryonics than proto-rationalists, and rationalist training probably makes you less likely to believe cryonics will work.

On the other hand, 0% of proto-rationalists had signed up for cryonics compared to 13% of experienced rationalists. 48% of proto-rationalists rejected the idea of signing up for cryonics entirely, compared to only 25% of experienced rationalists. So although rationalists are less likely to believe cryonics will work, they are much more likely to sign up for it. Last year's survey shows the same pattern.

This is not necessarily surprising. It only indicates that experienced rationalists and proto-rationalists treat their beliefs in different ways. Proto-rationalists form a belief, play with it in their heads, and then do whatever they were going to do anyway -  usually some variant on what everyone else does. Experienced rationalists form a belief, examine the consequences, and then act strategically to get what they want.

Imagine a lottery run by an incompetent official who accidentally sets it up so that the average payoff is far more than the average ticket price. For example, maybe the lottery sells only ten $1 tickets, but the jackpot is $1 million, so that each $1 ticket gives you a 10% chance of winning $1 million.

Goofus hears about the lottery and realizes that his expected gain from playing the lottery is $99,999. "Huh," he says, "the numbers say I could actually win money by playing this lottery. What an interesting mathematical curiosity!" Then he goes off and does something else, since everyone knows playing the lottery is what stupid people do.

Gallant hears about the lottery, performs the same calculation, and buys up all ten tickets.

The relevant difference between Goofus and Gallant is not skill at estimating the chances of winning the lottery. We can even change the problem so that Gallant is more aware of the unlikelihood of winning than Goofus - perhaps Goofus mistakenly believes there are only five tickets, and so Gallant's superior knowledge tells him that winning the lottery is even more unlikely than Goofus thinks. Gallant will still play, and Goofus will still pass.

The relevant difference is that Gallant knows how to take ideas seriously.

Taking ideas seriously isn't always smart. If you're the sort of person who falls for proofs that 1 = 2  , then refusing to take ideas seriously is a good way to avoid ending up actually believing that 1 = 2, and a generally excellent life choice.

On the other hand, progress depends on someone somewhere taking a new idea seriously, so it's nice to have people who can do that too. Helping people learn this skill and when to apply it is one goal of the rationalist movement.

In this case it seems to have been successful. Proto-rationalists think there is a 21% chance of a new technology making them immortal - surely an outcome as desirable as any lottery jackpot - consider it an interesting curiosity, and go do something else because only weirdos sign up for cryonics.

Experienced rationalists think there is a lower chance of cryonics working, but some of them decide that even a pretty low chance of immortality sounds pretty good, and act strategically on this belief.

This is not to either attack or defend the policy of assigning a non-negligible probability to cryonics working. This is meant to show only that the difference in cryonics status between proto-rationalists and experienced rationalists is based on meta-level cognitive skills in the latter whose desirability is orthogonal to the object-level question about cryonics.

(an earlier version of this article was posted on my blog last year; I have moved it here now that I have replicated the results with a second survey)

Since living in Oxford, one of the centers of the "effective altruism" movement, I've been spending a lot of time discussing the classic “effective altruism” topic -- where it would be best to focus our time and money.

Some people here seem to think that the most important thing we should be focusing our time and money on are speculative projects, or projects that promise a very high impact, but involve a lot of uncertainty.  One such very common example is "existential risk reduction", or attempts to make a long-term far future for humans more likely, say by reducing the chance of things that would cause human extinction.

I do agree that the far future is the most important thing to consider, by far (see papers by Nick Bostrom and Nick Beckstead).  And I do think we can influence the far future.  I just don't think we can do it in a reliable way.  All we have are guesses about what the far future will be like and guesses about how we can affect it. All of these ideas are unproven, speculative projects, and I don't think they deserve the main focus of our funding.

While I waffled in cause indecision for a while, I'm now going to resume donating to GiveWell's top charities, except when I have an opportunity to use a donation to learn more about impact.  Why?  My case is that speculative causes, or any cause with high uncertainty (reducing nonhuman animal suffering, reducing existential risk, etc.) requires that we rely on our commonsense to evaluate them with naīve cost-effectiveness calculations, and this is (1) demonstrably unreliable with a bad track record, (2) plays right into common biases, and (3) doesn’t make sense based on how we ideally make decisions.  While it’s unclear what long-term impact a donation to a GiveWell top charity will have, the near-term benefit is quite clear and worth investing in.

Focusing on Speculative Causes Requires Unreliable Commonsense

How can we reduce the chance of human extinction? It just makes sense that if we fund cultural exchange programs between the US and China, there will be more goodwill for the other within each country, and therefore the countries will be less likely to nuke each other. Since nuclear war would likely be very bad, it's of high value to fund cultural exchange programs, right?

Let's try another. The Machine Intelligence Research Institute (MIRI) thinks that someday artificial intelligent agents will become better than humans at making AIs. At this point, AI will build a smarter AI which will build an even smarter AI, and -- FOOM! -- we have a superintelligence. It's important that this superintelligence be programmed to be benevolent, or things will likely be very bad. And we can stop this bad event by funding MIRI to write more papers about AI, right?

Or how about this one? It seems like there will be challenges in the far future that will be very daunting, and if humanity handles them wrong, things will be very bad. But if people were better educated and had more resources, surely they'd be better at handling those problems, whatever they may be. Therefore we should focus on speeding up economic development, right?

These three examples are very common appeals to commonsense.  But commonsense hasn't worked very well in the domain of finding optimal causes.

Can You Pick the Winning Social Program?

Benjamin Todd makes this point well in "Social Interventions Gone Wrong", where he provides a quiz with eight social programs and asks readers to guess whether they succeeded or failed.

I'll wait for you to take the quiz first... doo doo doo... la la la...

Ok, welcome back. I don't know how well you did, but success on this quiz is very rare, and this poses problems for commonsense.  Sure, I'll grant you that Scared Straight sounds pretty suspicious. But the Even Start Family Literacy Program? It just makes sense that providing education to boost literacy skills and promote parent-child literacy activities should boost literacy rates, right? Unfortunately, it was wrong. Wrong in a very counter-intuitive way. There wasn't an effect.  

GiveWell and Commonsense's Track Record of Failure

Commonsense actually has a track record of failure. GiveWell has been talking about this for ages.  Every time GiveWell has found an intervention hyped by commonsense notions of high-impact and they've looked at it further, they've ended up disappointed.

The first was the Fred Hollows Foundation. A lot of people had been repeating the figure that the Fred Hollows Foundation could cure blindness for $50. But GiveWell found that number suspect.

The second was VillageReach. GiveWell originally put them as their top charity and estimated them as saving a life for under $1000. But further investigation kept leading them to revise their estimate until ultimately they weren't even sure if VillageReach had an impact at all.

Third, there is deworming. Originally, deworming was announced as saving a year of healthy life (DALY) for every $3.41 spent. But when GiveWell dove into the spreadsheets that resulted in that number, they found five errors. When the dust settled, the $3.41 figure was found to actually be off by a factor of 100. It was revised to $326.43.

Why shouldn't we expect this trend to not be the case in other areas where calculations are even looser and numbers are even less settled, like efforts devoted to speculative causes? Our only recourse is to fall back on interventions that are actually studied.

People Are Notoriously Bad At Predicting the (Far) Future

Cost-effectiveness estimates also frequently require making predictions about the future. Existential risk reduction, for example, requires making predictions about what will happen in the far future, and how your actions are likely to effect events hundreds of years down the road. Yet, experts are notoriously bad at making these kinds of predictions.

James Shanteau found in "Competence in Experts: The Role of Task Characteristics" (see also Kahneman and Klein's "Conditions for Intuitive Expertise: A Failure to Disagree") that experts perform well when thinking about static stimuli, thinking about things, and when there is feedback and objective analysis available. Furthermore, experts perform pretty badly when thinking about dynamic stimuli, thinking about behavior, and feedback and objective analysis are unavailable.

Predictions about existential risk reduction and the far future are firmly in the second category. So how can we trust our predictions about our impact on the far future? Our only recourse is to fall back on interventions that we can reliably predict, until we get better at prediction (or invest money in getting better at making predictions).

Even Broad Effects Require Specific Attempts

One potential resolution to this problem is to argue for “broad effects” rather than “specific attempts”.  Perhaps it’s difficult to know whether a particular intervention will go well or mistaken to focus entirely on Friendly AI, but surely if we improved incentives and norms in academic work to better advance human knowledge (meta-research), improved education, or advocated for effective altruism, the far future would be much better equipped to handle threats.

I agree that these broad effects would make the far future better and I agree that it’s possible to implement these broad effects and change the far future.  The problem, however, is it can’t be done in an easy or well understood way.  Any attempt to implement a broad effect would require a specific action that has an unknown expectation of success and unknown cost-effectiveness.  It’s definitely beneficial to advocate for effective altruism, but could this be done in a cost-effective way?  A way that’s more cost-effective at producing welfare than AMF?  How would you know?

In order to accomplish these broad effects, you’d need specific organizations and interventions to channel your time and money into.  And by picking these specific organizations and interventions, you’re losing the advantage of broad effects and tying yourself to particular things with poorly understood impact and no track record to evaluate. 

Focusing on Speculative Causes Plays Into Our Biases

We've now known for quite a long time that people are not all that rational. Instead, human thinking fails in very predictable and systematic ways.  Some of these ways make us less likely to take speculative causes seriously, such as ambiguity aversion, the absurdity heuristic, scope neglect, and overconfidence bias.

But there’s also a different side of the coin, with biases that might make people think badly about existential risk:

Optimism bias. People generally think things will turn out better than they actually will. This could lead people to think that their projects will have a higher impact than they actually will, which would lead to higher estimates of cost-effectiveness than is reasonable.

Control bias. People like to think they have more control over things than they actually do. This plausibly also includes control over the far future. Therefore, people are probably biased into thinking they have more control over the far future than they actually do, leading to higher estimates of ability to influence the future than is reasonable.

"Wow factor" bias. People seem attracted to more impressive claims. Saving a life for $2500 through a malaria bed net seems much more boring compared to the chance of saving the entire world by averting a global catastrophe. Within the Effective Altruist / LessWrong community, existential risk reduction is cool and high status, whereas averting global poverty is not. This might lead to more endorsement of existential risk reduction than is reasonable.

Conjunction fallacy.  People have a problem assessing probability properly when there are many steps involved, each of which has a chance of not happening. Ten steps, each with an independent 90% success rate, has only a 35% chance of success.  Focusing on the far future seems to involve that a lot of largely independent events happen the way that is predicted. This would mean people are worse at estimating their chances of helping the far future, creating higher cost-effectiveness estimates than is reasonable.

Selection bias.  When trying to find trends in history that are favorable for affecting the far future, some examples can be provided.  However, this is because we usually hear about the interventions that end up working, whereas all the failed attempts to influence the far future are never heard of again.  This creates a very skewed sample that can negatively bias our thinking about our success of influencing the far future.

It’s concerning there are numerous biases both weighted in favor and weighted against speculative causes, and this means we must tread carefully when assessing their merits.  However, I would strongly expect biases to be even worse in favor of speculative causes rather than against them, because speculative causes lack the available feedback and objective evidence needed to help insulate against bias, whereas a focus on global health does not.

Focusing on Speculative Causes Uses Bad Decision Theory

Furthermore, not only is the case for speculative causes undermined by a bad track record and possible cognitive biases, but the underlying decision theory seems suspect in a way that's difficult to place.         

Would you play a lottery with no stated odds?

Imagine another thought experiment -- you're asked to play a lottery. You have to pay $2 to play, but you have a chance at winning $100. Do you play?

Of course, you don't know, because you're not given odds. Rationally, it makes sense to play any lottery where you expect to come out ahead more often than not. If the lottery is a coin flip, it makes sense to pay $2 to have a 50/50 shot to win $100, since you'd expect to win $50 on average, and come ahead $48 each time. With a sufficiently high reward, even a one in a million chance is worth it. Pay $2 for a 1/1M chance of winning $1B, and you'd expect to come out ahead by $998 each time.

But $2 for the chance to win $100, without knowing what the chance is? Even if you had some sort of bounds, like you knew the odds had to be at least 1/150 and at most 1/10, though you could be off by a little bit. Would you accept that bet?

Such a bet seems intuitively uninviting to me, yet this is the bet that speculative causes offer me.

"Conservative Orders of Magnitude" Arguments

In response to these considerations, I've seen people endorsing speculative causes look at their calculations and remark that even if their estimate were off by 1000x, or three orders of magnitude, they still would be on solid ground for high impact, and there's no way they're actually off by three orders of magnitude. However, Nate Silver's The Signal and the Noise: Why So Many Predictions Fail — but Some Don't offers a cautionary tale:

Moody’s, for instance, went through a period of making ad hoc adjustments to its model in which it increased the default probability assigned to AAA-rated securities by 50 percent. That might seem like a very prudent attitude: surely a 50 percent buffer will suffice to account for any slack in one’s assumptions? It might have been fine had the potential for error in their forecasts been linear and arithmetic. But leverage, or investments financed by debt, can make the error in a forecast compound many times over, and introduces the potential of highly geometric and nonlinear mistakes.

Moody’s 50 percent adjustment was like applying sunscreen and claiming it protected you from a nuclear meltdown—wholly inadequate to the scale of the problem. It wasn’t just a possibility that their estimates of default risk could be 50 percent too low: they might just as easily have underestimated it by 500 percent or 5,000 percent. In practice, defaults were two hundred times more likely than the ratings agencies claimed, meaning that their model was off by a mere 20,000 percent.

Silver points out that when estimating how safe mortgage backed securities were, the difference between assuming defaults are perfectly uncorrelated and defaults are perfectly correlated is a difference of 160,000x in your risk estimate -- or five orders of magnitude.

If these kinds of five-orders-of-magnitude errors are possible in a realm that has actual feedback and is moderately understood, how do we know the estimates for cost-effectiveness are safe for speculative causes that are poorly understood and offer no feedback?  Again, our only recourse is to fall back on interventions that we can reliably predict, until we get better at prediction.

Value of Information, Exploring, and Exploiting

Of course, there still is one important aspect of this problem that has not been discussed -- value of information -- or the idea that sometimes it’s worth doing something just to learn more about how the world works.  This is important in effective altruism too, where we focus specifically on “giving to learn”, or using our resources to figure out more about the impact of various causes.

I think this is actually really important and is not a victim to any of my previous arguments, because we’re not talking about impact, but rather learning value.  Perhaps one could look to an "explore-exploit model", or the idea that we achieve the best outcome when we spend a lot of time exploring first (learning more about how to achieve better outcomes) before exploiting (focusing resources on achieving the best outcome we can).  Therefore, whenever we have an opportunity to “explore” further or learn more about what causes have high impact, we should take it.

Learning in Practice

Unfortunately, in practice, I think these opportunities are very rare.  Many organizations that I think are “promising” and worth funding further to see what their impact looks like do not have sufficiently good self-measurement in place to actually assess their impact or sufficient transparency to provide that information, therefore making it difficult to actually learn from them.  And on the other side of things, many very promising opportunities to learn more are already fully funded.  One must be careful to ensure that it’s actually one’s marginal dollar that is getting marginal information.

The Typical Donor

Additionally, I don’t think the typical donor is in a very good position to assess where there is high value of information or have the time and knowledge to act upon this information once it is acquired.  I think there’s a good argument for people in the “effective altruist” movement to perhaps make small investments in EA organizations and encourage transparency and good measurement in their operations to see if they’re successfully doing what they claim (or potentially create an EA startup themselves to see if it would work, though this carries large risks of further splitting the resources of the movement).

But even that would take a very savvy and involved effective altruist to pull off.  Assessing the value of information on more massive investments like large-scale research or innovation efforts would be significantly more difficult, beyond the talent and resources of nearly all effective altruists, and are probably left to full-time foundations or subject-matter experts.

GiveWell’s Top Charities Also Have High Value of Information

As Luke Muehlhauser mentions in "Start Under the Streetlight, Then Push Into the Shadows", lots of lessons can be learned only by focusing on the easiest causes first, even if we have strong theoretical reasons to expect that they won’t end up being the highest impact causes once we have more complete knowledge.

We can use global health cost-effectiveness considerations as practice for slowly and carefully moving into the more complex and less understood domains.  There even are some very natural transitions, such as beginning to look at "flow through effects" of reducing disease in the third-world and beginning to look at how more esoteric things affect the disease burden, like climate change.  Therefore, even additional funding for GiveWell’s top charities has high value of information.  And notably, GiveWell is beginning this "push" through GiveWell Labs.

Conclusion

The bottom line is that sometimes things look too good to be true.  Therefore, I should expect that the actual impact of speculative causes that make large promises, upon a thorough investigation, will be much lower.

And this has been true in other domains. People are notoriously bad at estimating the effects of causes in both the developed world and developing world, and those are the causes that are near to us, provide us with feedback, and are easy to predict. Yet, from the Even Start Family Literacy Program to deworming estimates, our commonsense has failed us.

Add to that the fact that we should expect ourselves to perform even worse at predicting the far future. Add to that optimism bias, control bias, "wow factor" bias, and the conjunction fallacy, which make it difficult for us to think realistically about speculative causes. And then add to that considerations in decision theory, and whether we would bet on a lottery with no stated odds.

When all is said and done, I'm very skeptical of speculative projects.  Therefore, I think we should be focused on exploring and exploiting.  We should do whatever we can to fund projects aimed at learning more, when those are available, but be careful to make sure they actually have learning value.  And when exploring isn’t available, we should exploit what opportunities we have and fund proven interventions.

But don’t confuse these two concepts and fund causes intended for learning because of their actual impact value.  I’m skeptical about these causes actually being high impact, though I’m open to the idea that they might be and look forward to funding them in the future when they become better proven.     

-

Followed up in: "What Would It Take To 'Prove' A Skeptical Cause" and "Where I've Changed My Mind on My Approach to Speculative Causes".

This was also cross-posted to my blog and to effective-altruism.com.

I'd like to thank Nick Beckstead, Joey Savoie, Xio Kikauka, Carl Shulman, Ryan Carey,  Tom Ash, Pablo Stafforini, Eliezer Yudkowsky, and Ben Hoskin for providing feedback on this essay, even if some of them might strongly disagree with it's conclusion.

Related to: Politics is the Mind-Killer, Keep Your Identity Small

Followed By: How to Not Get Offended

One oft-underestimated threat to epistemic rationality is getting offended. While getting offended by something sometimes feels good and can help you assert moral superiority, in most cases it doesn't help you figure out what the world looks like. In fact, getting offended usually makes it harder to figure out what the world looks like, since it means you won't be evaluating evidence very well. In Politics is the Mind-Killer, Eliezer writes that "people who would be level-headed about evenhandedly weighing all sides of an issue in their professional life as scientists, can suddenly turn into slogan-chanting zombies when there's a Blue or Green position on an issue." Don't let yourself become one of those zombies-- all of your skills, training, and useful habits can be shut down when your brain kicks into offended mode!

One might point out that getting offended is a two-way street and that it might be more appropriate to make a post called "Don't Be Offensive." That feels like a just thing to say-- as if you are targeting the aggressor rather than the victim. And on a certain level, it's true-- you shouldn't try to offend people, and if you do in the course of a normal conversation it's probably your fault. But you can't always rely on others around you being able to avoid doing this. After all, what's offensive to one person may not be so to another, and they may end up offending you by mistake. And even in those unpleasant cases when you are interacting with people who are deliberately trying to offend you, isn't staying calm desirable anyway?

The other problem I have with the concept of being offended as victimization is that, when you find yourself getting offended, you may be a victim, but you're being victimized by yourself. Again, that's not to say that offending people on purpose is acceptable-- it obviously isn't. But you're the one who gets to decide whether or not to be offended by something. If you find yourself getting offended to things as an automatic reaction, you should seriously evaluate why that is your response.

There is nothing inherent in a set of words that makes them offensive or inoffensive-- your reaction is an internal, personal process. I've seen some people stay cool in the face of others literally screaming racial slurs in their faces and I've seen other people get offended by the slightest implication or slip of the tongue. What type of reaction you have is largely up to you, and if you don't like your current reactions you can train better ones-- this is a core principle of the extremely useful philosophy known as Stoicism.

Of course, one (perhaps Robin Hanson) might also point out that getting offended can be socially useful. While true-- quickly responding in an offended fashion can be a strong signal of your commitment to group identity and values^[1]-- that doesn't really relate to what this post is talking about. This post is talking about the best way to acquire correct beliefs, not the best way to manipulate people. And while getting offended can be a very effective way to manipulate people-- and hence a tactic that is unfortunately often reinforced-- it is usually actively detrimental for acquiring correct beliefs. Besides, the signalling value of offense should be no excuse for not knowing how not to be offended. After all, if you find it socially necessary to pretend that you are offended, doing so is not exactly difficult.

Personally, I have found that the cognitive effort required to build a habit of not getting offended pays immense dividends. Getting offended tends to shut down other mental processes and constrain you in ways that are often undesirable. In many situations, misunderstandings and arguments can be diminished or avoided completely if one is unwilling to become offended and practiced in the art of avoiding offense. Further, some of those situations are ones in which thinking clearly is very important indeed! All in all, while getting offended does often feel good (in a certain crude way), it is a reaction that I have no regrets about relinquishing.

[1] In Keep Your Identity Small, Paul Graham rightly points out that one way to prevent yourself from getting offended is to let as few things into your identity as possible.