Newest Submissions - Less Wrong

Some reservations about Singer's child-in-the-pond argument

Wed, 19 Jun 2013 23:54:58 +0000

Submitted by JonahSinick • 9 votes • 5 comments

Peter Singer is one of the most influential philosophers, and is a strong candidate for being the person who has helped the effective altruist community the most.

In the past, Peter Singer often argued that [the moral obligation to rush into a shallow pond to save a drowning child at the cost of ruining one’s shoes] is equivalent to [the moral obligation to give to charities that reduce extreme poverty]. For example, in this 2009 video he said:

Imagine that you’re walking across a shallow pond and you notice that a small child has fallen in, and is in danger of drowning […] Of course, you think you must rush in to save the child. Then you remember that you’re wearing your favorite, quite expensive, pair of shoes and they’ll get ruined if you rush into the pond. Is that a reason for not saving the child? I’m sure you’ll say no it isn’t, you just can’t compare the life of a child to the cost of a pair of shoes, no matter how expensive. […] But think about how that relates to your situation in the world today. There are children whose lives you can save […] Nearly 10 million children die every year from avoidable, poverty related causes. And it wouldn’t take a lot to save the lives of these children. We can do it. For the cost of a pair of shoes, perhaps, you could save the life of a child. […] There’s some luxury that you could do without. And with that money, you could give to an organization to reduce extreme poverty in the world, and save lives of children. […] I think that this is what we ought to be doing.

Since Singer first posed the analogy, new information and understanding has emerged, which cast doubt on the relevance of Singer’s analogy. Singer used a different analogy in his recent TED talk (in which he discussed the death of Wang Yue), but whether explicitly or implicitly, the “child in a pond” meme has caught on. In light of recent developments, it’s important to highlight the fact that the opportunities to donate to alleviate global in our present world are disanalogous to the opportunity in Singer’s “child in a pond” scenario.

The most expensive pair of shoes that I own costs ~$120, and I would guess that the average American doesn’t own a pair of shoes that costs more than $200. With this in mind, Singer’s analogy suggests that one can save a life of a child in the developing world for less than $200. To determine whether Peter Singer’s analogy is a good one, we need to examine the empirical data concerning the cost of saving a child’s life.

GiveWell spent five years looking for outstanding charities that alleviate poverty in the developing world. GiveWell's current top recommended charity, Against Malaria Foundation (AMF), distributes long lasting insecticide treated nets to guard recipients against malaria. GiveWell’s explicit estimate of AMF’s cost per life saved is just under $2,300. The cost of bed nets has recently fallen, and this is expected to decrease AMF’s cost per life saved, but not by a large margin.

GiveWell Co-Executive Director Holden Karnofsky has written about how explicit expected value estimates shouldn’t be taken literally, and in particular, that explicit estimates of the value of philanthropic opportunities should be adjusted to account for one’s Bayesian prior over the effectiveness of all philanthropic opportunities. In June 2012, GiveWell senior research analyst Alexander Berger wrote(speaking for himself rather than for GiveWell)

I don't think the expected value of a $1600 donation to AMF [an earlier cost-effectiveness estimate for AMF's cost per life saved] is actually anywhere near one life saved. The reason for this has nothing to do with how AMF works and is more a feature of its place in the total distribution of charity cost-effectiveness. I think there are a variety of practices in cost-effectiveness estimation that push in favor of a difficult-to-estimate positive bias (e.g. using evidence from RCTs, which are generally conducted in the most promising circumstances), that the most extreme cost-effectiveness estimates are more likely to be biased, and that the benefit of a marginal contribution is almost always less than the benefit of an average contribution. All of these conspire to make me think that the estimate that GiveWell provides for the "cost-per-life saved" for AMF is not the correct number for estimating the expected value of a contribution to AMF.

In the section “Concrete factors that further reduce the expected value of donating to AMF” of my blog post Robustness of Cost-Effectiveness Estimates and Philanthropy, I listed eleven concrete factors that increase AMF’s expected cost per life saved.

The reason why saving the child drowning in a pond in Singer’s hypothetical is obviously the right thing to do is that the personal cost associated with doing so is negligible relative to the benefit to others. The cost of saving a life in the developing world by donating to AMF is at least 10x greater than the cost in Singer’s “child in a pond” analogy, and possibly much greater. This substantially weakens Singer’s argument.

I raised this point in a recent comment thread on the GiveWell blog, and Doug S. concurred, writing

Honestly, there really is a big difference to me if X is different by orders of magnitude. The U.S. federal minimum wage is currently $7.25 an hour. Payroll taxes are 7.5%, so take-home pay becomes $6.70 an hour. It takes 343 hours – two months, working full time – working a minimum wage job to earn the $2300 it takes your #1 charity to save a life. There’s a big difference between $200 and $2000, between one week of minimum wage work and two months of minimum wage work.

Holden responded:

I think Jonah and Doug are both looking for more precision than is reasonable. Robust facts about disparities in wealth – which you will also see qualitatively if you travel to the developing world – are sufficient to make the point that you have a great deal of power to help others a lot by giving up a little. If you’re looking for any sort of precise “dollar cost per quantity of good accomplished” (over and above the kind of robust comparisons I just described) such that a factor of 5-10 is crucial to how much you decide to give, I think it is – and long knowably has been – unrealistic to get such a thing. I think nearly all targets of Peter Singer’s argument have long implicitly recognized this fact. Perhaps there are some arguments for which such precision would be necessary, but if so they aren’t arguments that I see as having much traction. I don’t empathize with the view that such precision is necessary in order to make the broad argument that you ought to give generously.

What Holden’s comment misses is that there’s a big difference between the following two statements:

(1) “A rough estimate for the cost of saving a life is the cost of an expensive pair of shoes, but it could be much higher or much lower”

(2) “A rough estimate for the cost of saving a life is over 10x greater than the cost of an expensive pair of shoes, but the cost is probably higher, and possibly much higher, due to Bayesian regression.”

The problem with Singer’s “child in a pond” analogy isn’t that real world cost-effectiveness estimates aren’t precise. The problem with Singer’s “child in a pond” analogy is that there’s a strong case for the cost-effectiveness of donating to AMF being vastly lower than Singer’s analogy suggests.

Peter Singer has been very successful in getting people interested in donating to alleviate global poverty. One could argue that his “child in a pond” contributed to his success, and that continuing to use it is, for this reason, justified. Nevertheless, the analogy is problematic.

Vipul Naik wrote (paraphrased):

There is a tension between the tactically optimal approach for convincing a larger number of people to donate more, and the argument that is most grounded in empirical reality. I think that rather than minimizing the tension, it’s more courageous and epistemically admirable to openly and very explicitly admit that Singer-style (implicit or explicit) “you-can-save-a-life-for-the-price-of-a-pair-of-shoes” *if true*, would be far more compelling a reason to donate than the argument based on disparities in wealth.

See also this comment where Carl Shulman writes: “I think it’s bad news for probably mistaken estimates to spread, and then disillusion the readers or make the writers look biased. If people interested in effective philanthropy go around trumpeting likely wrong (over-optimistic) figures and don’t correct them, then the community’s credibility will fall, and bad models and epistemic practices may be strengthened.”

Singer's argument is not the only argument for donating to alleviate poverty in the developing world. For example, in a recent blog post, Holden wrote:

To us, the strongest form of the challenge [to donate to alleviate poverty in the developing world] is not “How much should I give when $X saves a life?” but “How much should I give, knowing that I have massive wealth compared to the global poor?” Perhaps the most vivid illustration comes not from Against Malaria Foundation (our #1-rated charity) but from GiveDirectly (our #2). If you give $1000 to GiveDirectly, ~$900 will end up in the hands of people whose resources are a tiny fraction of yours. GiveDirectly’s estimate – which we believe is less sensitive to guesswork than “cost per life saved” figures – is that recipients live on ~65 cents per day, implying that such a donation could roughly double the annual consumption for a family of four, not counting any long term benefits.

As Vipul commented, this argument is much weaker than Singer’s “child in a pond” argument.

In Living High and Letting Die: Our Illusion of Innocence (pg. 135) Peter Unger gives a Singer-style analogy that can be made more faithful to present day empirical realities than Singer’s “child in a pond” analogy. The form of the argument (modified for use in the present context) is this:

Imagine that you have a car that's worth AMF’s actual cost per life saved. You park your car on unused train tracks and get out in order to walk around. You see a child playing in a tunnel off in the distance, and see a train headed toward the tunnel. If the train proceeds, the train will kill the child. You have access to a switch that can be used to divert the train toward the unused train tracks where your car is parked. If you flip the switch, the train will demolish your car, but nobody will be killed. Do you flip the switch?

I think that most people would say that flipping the switch is the right thing to do. But I don’t think that they would say that the moral obligation is as great as the moral obligation in Singer's “child in a pond” scenario.

Acknowledgments: Thanks to Vipul Naik, Nick Beckstead and Luke Muehlhauser for helpful feedback on an earlier version of this post.

Note: I formerly worked as a research analyst at GiveWell. All views are my own.

5 comments

Giving Now Currently Seems to Beat Giving Later

Wed, 19 Jun 2013 18:40:52 +0000

Submitted by peter_hurford • 0 votes • 6 comments

Abstract: There is a debate between either donating now or donating later (investing, realizing the returns of one's investment, and donating in a lumpsum upon death). While donating later may be appropriate in some circumstances, right now we have the ability to donate now in order to go meta and recruit future donors who otherwise wouldn't have donated, thus adding more money than returns on investment.

Introduction

Among those people interested in doing as much as they can to make the world a better place and think donating their money as effectively as possible is a good way to do that, there is a debate about whether one should either (a) donate a specific portion of one's income in small installments each year or (b) invest one's income and then donate as much as possible in one large lump sum right before death. Of course, there are other positions -- like donate every month or donate every decade, but in a very relevant sense the debate is between two options: "give now" vs. "give later".

In this essay, I will defend the "give now" camp and rebut the "give later" camp, thus explaining why I will continue to donate once a year, though think any sort of regular time period smaller than each year is probably also acceptable.

But First, Why Give Later?

The biggest champ for the "give later" crowd is Robin Hanson, who makes his view known most recently in "If More Now, Less Later":

But at the margin, a person who saves another dollar, or chooses not to borrow another dollar, must typically expect the financial returns from their investments will help them more in the future than will such indirect effects of spending today. In fact, they should expect this savings will benefit their future self more than any of these other ways of spending today. After all, why give up money today if that both gives you less to spend today, and gives you less in the future? So there wouldn’t be any savings, or less than maximal borrowing, if people didn’t expect more gains later from saving than from spending today.

This implies that unless charity recipients are saving nothing and borrowing as much as they possibly can, they must expect that you would benefit them more in the future by saving and giving them the returns of your savings later, than if you had given them the money today, even after taking into account all of the ways in which their spending today might help them in the future. So there really must be a tradeoff between helping today and helping later; if you help more today, you help less in the future. At least if you help them in a way they could have helped themselves, if only they had the money.

Or, more basically, there are two concepts:

First, there is the "growth rate" of donation -- by donating to a non-profit now, they get to do things with your money now, helping people immediately who then could be in a better position to give back, or by spending money on dreaded "overhead" (which isn't evil, by the way, but that's a matter for another time), putting them in a better position to grow now.

Second, there is the "investment rate" of saving/investing your donation money rather than donating now. This should be the easier of the two concepts -- you save your money, it grows in a certain percentage, and then right before you're about to die, you take all that money out and donate it.

And then to oversimpify the complex topic, you should be willing to donate now as long as you think "growth rate > investment rate", and save now as long as you think "investment rate > growth rate", plus a few other complications to be discussed.

Ok, So What's The Investment Rate, Then?

Finding out the precise investment rate is complicated, but I think Jeff Kaufman has done some really good work on it in "What Rates of Return Should You Expect?". Basically, many sites suggest returns on investment between 6% and 12%, but this (a) ignores the average of 3% inflation, (b) involves cherry-picking, and (c) only focuses on US data. Articles like The Economist's "Beware of the Bias" point this out.

So if we're really optimistic, we could consider the investment rate to be 12% and if we're really pessimistic, we could consider it to be 0%, or lower! But I'd expect the real (inflation adjusted) rate of return on your investment to be between 3 and 5%. This means that if I had opted to instead take the $659.70 donation I made last year and instead invested in it, given that I expect to live around 70 more years, I could realistically realize $5,372.94 to $21,687.36 if compounded monthly.

Ok, So What's The Growth Rate, Then?

But what about the other side of the coin -- growth rate from your donation? I see the potential size of this being huge. Unlike the investment rate which I think is overestimated, the growth rate seems underestimated.

Honestly, whatever the growth rate is on your donation is going to depend largely and entirely on where you donate. But if you're donating to advocacy, you could realize really large returns because of what's called the haste consideration:

[I]magine two worlds:

(1) You don’t do anything altruistic for the next 2 years and then you spend the rest of your life after that improving the world as much as you can.

(2) You spend the next 2 years influencing people to become effective altruists and convince one person who is at least as effective as you are at improving the world. (And assume that this person wouldn’t have done anything altruistic otherwise.) You do nothing altruistic after the next 2 years, but the person you convinced does at least as much good as you did in (1).

By stipulation, world (2) is improved at least as much as world (1) is because, in (2), the person you convinced does at least as much good as you did in (1).

Essentially, convincing just one person to do what you would do (join Giving What We Can, donate to existential risk, do anti-aging research, etc.) as ardently as you would do it (or nearly as much) would have the same effect as your entire life's work. Robin Hanson stated the case for giving later over giving now as "if more now, less later".

But with the haste consideration, it's actually if more now, more later, because you'll be convincing people to do things they would otherwise have not done at all (presumably).

And you can donate money to convincing people to do those things. A donation to Giving What We Can can fund their media outreach and recruit more people to the idea of donating effectively. A donation to MIRI could fund more salaries and recruitment efforts, etc.

Thus, let's assume that, in my lifetime, I'll earn at least a $50K/year salary. Now assume that it costs a conservative $10K/year to recruit a Giving What We Can lifetime, committed member, who will also earn at least a $50K/year salary through outreach. (Feel free to quibble with these assumptions if you'd like, but don't risk missing the main point.)

Now consider that I can either:

(1) invest 10% of my salary each year at 5% return compounded monthly, and then, upon my death, "buy" as many committed GWWC members as possible.

(2) donate 10% of my salary each year to buy as many committed GWWC members as possible.

According to this investment calculator (which I used earlier), option #1 will yield $3,200,782.84, which could buy 320 committed GWWC members.

Option #2 would add a committed member every two years, producing 45 committed GWWC members over my predicted 70 year lifespan. This looks bad, but now remember that each of those members would then get a chance to recruit new members using their money. The first person I get would then commit $340K of their own lifetime earnings over the remaining period (68 years, since I recruited them on year 2, at $5K/year contribution). I'd only need to recruit ten members myself to dwarf the investment strategy, and that's not even counting all the second-stage members those first-stage members recruit.

At this point, the group effectively doubles over the two year period. I save up to $10K at year two and recruit someone, and then in two years (year #4) we both have $10K and recruit someone, and then in two more years (year #6) all four of us recruit have $10K and recruit someone each, bringing the total amount of people recruited to twelve. By year #70, I will have recruited, directly and indirectly, 1+2^34 people, or 17.1 billion new GWWC members.

Obviously, this is implausible because there aren't even 17 billion people and recruitment efforts would certainly be nonlinear. And there are other problems. But it's a simple way to prove the point. Hopefully this shows the benefit of compounding via the haste consideration to recruit new people who wouldn't have donated their money otherwise.

Even if Option #1 were considered to recruit people who then saved all their money and recruited other people, by year #140 it would only have 102,400 people recruited, whereas Option #2 would have roughly 10^41 people recruited.

While the growth rate probably isn't 100% every two years, it's considerably higher than 5%, I think. Thus since growth rate > interest rate, I advocate donating now. At least until someone finds the inevitable unobvious flaw in my thinking.

Of course, we'll probably hit diminishing marginal returns on GWWC recruiting soon, and thus should reconsider our donation target, re-evaluate the growth rate, and maybe shift back to investing rather than donating in the future. But, right now, the prospects are still high and there are still many bright and active individuals who have not yet been recruited. Right now, we can use our money to make sure they use their money better.

Thus my answer to Robin Hanson: More (of our money) now, more (of their money) later. At least, while it lasts.

Other Complications

Now time to explore some additional factors that may modify the consideration. The main argument has been made and this is is basically appendix material. Feel free to stop reading at this point, if you wish.

US Tax Law

This is an obvious factor to consider, but US law says that if you donate instead of invest, you can claim a deduction. This deduction would appear annually, thus allowing you to donate additional money. If you're earning the 50K I was talking about earlier, you'd be taxed at effectively 16.9%. This would mean you could get back $845 on a $5K donation, assuming I understand tax law right.

If I donate $5K annually without investing, I'd actually be able to donate $409,150 over my seventy years rather than the original $350,000. If I had invested and then donated it all, I'd get $3,741,715.14 instead of $3,200,782.84.

The Changing Nature of Non-Profits

There's a good chance that if Giving What We Can stopped receiving donations, it would just collapse and then the opportunity to donate to it in seventy years would be gone (though I suppose it could be refounded with the new cash). Additionally, GWWC couldn't learn from the seventy years of being able to operate, spend money, and try things.

And GWWC seems to be in a prime place right now to expand a lot and really grow it's outreach. Thus it's at a critical time to fund when the marginal cost to recruit a new member is at probably some of the lowest it will be. Now, not in seventy years, is a great time to get in.

Furthermore, I speculate that the marginal value will grow at a rate higher than the interest, such that $5K/year for seventy years would buy more members for GWWC than a flat $3.7 million in seventy years would.

You Might Be Smarter In The Future

There's a good chance there might be better giving opportunities in seventy years, identified through our collective greater intelligence, your years of added experience, and the changing world (though GiveWell disagrees, as does GWWC). This could mean it is better to give later, because you could be donating to an overall higher impact opportunity.

You Might Change Your Values

Some people are concerned that if they stored their donation money for seventy years, their future self might not care to donate it anymore. Personally, the way I view my personal identity, I'd rather have done what my future self wanted me to do, so if my future self seventy years from now wished I would have been less altruistic, I want to be less altruistic now, so I don't see this as a concern. But if you do, you should donate now, before your future self has a chance to redirect the money.

One way to protect from this might be to use a donor advised fund, which would also get the benefit of more charity deductions (since you can deduct each contribution to your fund) and your future wisdom while forcing your future self to donate.

Better Advocacy

It seems much easier to convince someone to give a little each year than to follow a solid savings plan -- as Scott Alexander states, there are some strange psychological effects that make us feel strongly like giving now and giving regularly is better, and we can secure the warm fuzzies feelings and social status of being considered a regular donor.

Moreover, if you convinced a friend to donate, you'd know much more immediately if they actually plan on sticking to it if you can see a donation they make in December rather than having to wait until they die. (Though I suppose this too could be solved by having them contribute to your donor advised fund, if they trust you a lot. But I imagine the average person wouldn't.)

Thus even if privately you should save, I think the best strategy to adopt publicly (when not speaking from an analytical point-of-view) is to encourage others to donate now.

Also cross-posted on my blog, on Felicifia, and on the Effective Altruist blog.

6 comments

Weekly LW Meetups

Fri, 14 Jun 2013 15:02:32 +0000

Submitted by FrankAdamek • 1 votes • 0 comments

Irregularly scheduled Less Wrong meetups are taking place in:

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Locations with regularly scheduled meetups: Austin, Berkeley, Cambridge, MA, Cambridge UK, Madison WI, Melbourne, Mountain View, New York, Ohio, Portland, Salt Lake City, Seattle, Toronto, Vienna, Waterloo, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

If you missed the deadline and wish to have your meetup featured, you can reach me on gmail at frank dot c dot adamek.

In addition to the handy sidebar of upcoming meetups, a meetup overview will continue to be posted on the front page every Friday. These will be an attempt to collect information on all the meetups happening in the next weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll now also have the benefit of having your meetup mentioned in a weekly overview. These overview posts will be moved to the discussion section when the new post goes up.

Please note that for your meetup to appear in the weekly meetups feature, you need to post your meetup before the Friday before your meetup!

If you check Less Wrong irregularly, consider subscribing to one or more city-specific mailing list in order to be notified when an irregular meetup is happening: Atlanta, Berlin, Chicago, Helsinki, London, Marin CA, Ottawa, Pittsburgh, Southern California (Los Angeles/Orange County area), St. Louis, Vancouver, Washington, DC.

Whether or not there's currently a meetup in your area, you can sign up to be notified automatically of any future meetups. And if you're not interested in notifications you can still enter your approximate location, which will let meetup-starting heroes know that there's an interested LW population in their city!

If your meetup has a mailing list that you'd like mentioned here, or has become regular and isn't listed as such, let me know!

Want to help out the common good? If one of the meetups listed as regular has become inactive, let me know so we can present more accurate information to newcomers.

0 comments

Effective Altruism Through Advertising Vegetarianism?

Wed, 12 Jun 2013 18:50:31 +0000

Submitted by peter_hurford • 11 votes • 466 comments

Abstract: If you value the welfare of nonhuman animals from a consequentialist perspective, there is a lot of potential for reducing suffering by funding the persuasion of people to go vegetarian through either online ads or pamphlets. In this essay, I develop a calculator for people to come up with their own estimates, and I personally come up with a cost-effectiveness estimate of $0.02 to $65.92 needed to avert a year of suffering in a factory farm. I then discuss the methodological criticism that merits skepticism of this estimate and conclude by suggesting (1) a guarded approach of putting in just enough money to help the organizations learn and (2) the need for more studies should be developed that explore advertising vegetarianism in a wide variety of media in a wide variety of ways, that include decent control groups.

Introduction

I start with the claim that it's good for people to eat less meat, whether they become vegetarian -- or, better yet, vegan -- because this means less nonhuman animals are being painfully factory farmed. I've defended this claim previously in my essay "Why Eat Less Meat?". I recognize that some people, even those who consider themselves effective altruists, do not value the well-being of nonhuman animals. For them, I hope this essay is interesting, but I admit it will be a lot less relevant.

The second idea is that it shouldn't matter who is eating less meat. As long as less meat is being eaten, less animals will be farmed, and this is a good thing. Therefore, we should try to get other people to also try and eat less meat.

The third idea is that it also doesn't matter who is doing the convincing. Therefore, instead of convincing our own friends and family, we can pay other people to convince people to eat less meat. And this is exactly what organizations like Vegan Outreach and The Humane League are doing. With a certain amount of money, one can hire someone to distribute pamphlets to other people or put advertisements on the internet, and some percentage of people who receive the pamphlets or see the ads will go on to eat less meat. This idea and the previous one should be uncontroversial for consequentialists.

But the fourth idea is the complication. I want my philanthropic dollars to go as far as possible, so as to help as much as possible. Therefore, it becomes very important to try and figure out how much money it takes to get people to eat less meat, so I can compare this to other estimations and see what gets me the best "bang for my buck".

Other Estimations

I have seen other estimates floating around the internet that try to estimate the cost of distributing pamphlets, how many conversions each pamphlet produces, and how much less meat is ate via each conversion. Brian Tomasik calculates $0.02 to $3.65 [PDF] per year of nonhuman animal suffering prevented, later $2.97 per year, and then later $0.55 to $3.65 per year.

Jess Whittlestone provides statistics that reveal an estimate of less than a penny per year[1].

Effective Animal Activism, a non-profit evaluator for animal welfare charities, came up with an estimate [Excel Document] of $0.04 to $16.60 per year of suffering averted, that also takes into account a variety of additional variables, like product elasticity.

Jeff Kaufman uses a different line of reasoning, by estimating how many vegetarians there are and guessing how many of them came via pamphlets, estimates it would take $4.29 to $536 to make someone vegetarian for one year. Extrapolating from that using at a rate of 255 animals saved per year and a weighted average of 329.6 days lived per animal (see below for justification of both assumptions), would give $0.02 to $1.90 per year of suffering averted[2].

A third line of reasoning, also by Jeff Kaufman, was to measure the amount of comments on the pro-vegetarian websites advertised in these campaigns and found that 2-22% of them were about an intended behavior change (eating less meat, going vegetarian, or going vegan), depending on the website. I don't think we can draw any conclusions from this, but it's interesting.

To make my calculations, I decided to make a calculator. Unfortunately, I can't embed it here, so you'd have to open it in a new tab as a companion piece.

I'm going to start by using the following formula: Years of Suffering Averted per Dollar = (Pamphlets / dollar) * (Conversions / pamphlet) * (Veg years / conversion) * (Animals saved / veg year) * (Days lived / animal)

Now, to get estimations for these variables.

Pamphlets Per Dollar

How much does it cost to place the advertisement, whether it be the paper pamphlet or a Facebook advertisement? Nick Cooney, head of the Humane League, says the cost-per-click of Facebook ads is 20 cents.

But what about the cost per pamphlet? This is more of a guess, but I'm going to go with <a href="">Vegan Outreach's suggested donation of $0.13 per "Compassionate choices" booklet.

However, it's important to note that this cost must also include opportunity cost -- leafleters must forego the ability to use that time to work a job. This means I must include an opportunity cost of say $8/hr on top of that, making the actual cost $0.27 assuming a pamphlet is given out each minute of volunteer time, meaning 3.7 people are reached per dollar from pamphlets. For Facebook advertisements, the opportunity cost is trivial.

Conversions Per Pamphlet

This is the estimate with the biggest target on it's head, so to speak. How many people do we get to actually change their behavior with a simple pamphlet or Facebook advertisement? Right now, we have three lines of evidence:

Facebook Study

Humane League did A $5000 Facebook advertisement campaign. They bought ads that look like this...

...and sent people to websites (like this one or this one) with auto-playing videos that start playing and show the horrors of factory farming.

Afterward, there was another advertisement run to people who "liked" the video page, offering a 1 in 10 chance of winning a free movie ticket in order to take a survey. Everyone who emailed in asking for a free vegetarian starter kit were also emailed a survey. 104 people took the survey and there were 32 reported vegetarians[3] and 45 people reported, for example, that their chicken consumption decreased "slightly" or "significantly".

7% of visitors liked the page and 1.5% of visitors ordered a starter kit. Assuming all the other people went away from the video not changing their consumption, this survey would lead us to (very tenuously) think about 2.6% of people seeing the video will become a vegetarian[4].

(Here's the results of the survey in PDF.)

Pamphlet Study

A second study discussed in "The Powerful Impact of College Leafleting (Part 1)" and "The Powerful Impact of College Leafleting: Additional Findings and Details (Part 2)" looked specifically at pamphlets.

Here, Humane League staff visited two large East Coast state schools and distributed leaflets. They then returned two months later and surveyed people walking by. Those who remember receiving a leaflet earlier were counted. They found about 2% of those receiving a pamphlet went vegetarian.

Vegetarian Years Per Conversion

But once a pamphlet or Facebook advertisement captures someone, how long will they stay vegetarian? One survey showed vegetarians refrain from eating meat for an average of 6 years or more. Another study I found says 93% of vegetarians stay vegetarian for at least three years.

Animals Saved Per Vegetarian Year

And once you have a vegetarian, how many animals do they save per year? CountingAnimals says 406 animals saved per year.

The Humane League suggests 28 chickens, 2 egg industry hens, 1/8 beef cow, 1/2 pig, 1 turkey, and 1/30 dairy cow per year (total = 31.66 animals), and does not provide statistics on fish. This agrees with CountingAnimals on non-fish totals.

Days Lived Per Animal

One problem, however, is that saving a cow that could suffer for years is different from saving a chicken that suffers for only about a month. Using data from Farm Sanctuary plus World Society for the Protection of Animals data on fish [PDF], I get this table:

Animal	Number	Days Alive
Chicken (Meat)	28	42
Chicken (Egg)	2	365
Cow (Beef)	0.125	365
Cow (Milk)	0.033	1460
Fish	225	365

This makes the weighted average 329.6 days[5].

Accounting For Biases

As I said before, our formula was Years of Suffering Averted = (Pamphlets / dollar) * (Conversions / pamphlet) * (Veg years / conversion) * (Animals saved / veg year) * (Days lived / animal).

Let's plug these values in... Years of Suffering Averted per Dollar = 5 * 0.02 * 3 * 255.16 * 329.6/365 = 69.12.

Or, assuming all this is right (and that's a big assumption), it would cost less than 2 cents to prevent a year of suffering on a factory farm by buying vegetarians.

I don't want to make it sound like I'm beholden to this cost estimate or that this estimate is the "end all, be all" of vegan outreach. Indeed, I share many of the skepticisms that have been expressed by others. The simple calculation is... well... simple, and it needs some "beefing up", no pun intended. Therefore, I also built a "complex calculator" that works on a much more complex formula[6] that is hopefully correct[7] and will provide a more accurate estimation.

The big, big deal for the surveys is concern for bias. The most frequently mentioned bias is social desirability bias, or people who say they reduced meat just because they want to please the surveyor or look like a good person, which actually happens a lot more on surveys than we'd like.

To account for this, we'll have to figure out how inflated answers are because of this bias and then scale the answers down by that amount. Nick Cooney who says that he's been reading studies that about 25% to 50% of people who say they are vegetarian actually are, though I don't yet have the citations. Thus, if we find out that an advertisement creates two meat reducers, we'd scale that down to one reducer if we're expecting a 50% desirability bias.

The second bias that will be a problem for us is non-response bias, as those who don't reduce their diet are less likely to take the survey and therefore less likely to be counted. This is especially true in the Facebook study, which only measures people who "liked" or requested a starter kit, showing some pro-vegetarian affiliation.

We can balance this out by assuming everyone who didn't take the survey went on to have no behavior change whatsoever. Nick Cooney's Facebook Ad Survey is for the 7% of people who liked the page (and then responded to the survey), and obviously those who liked the page are more likely to reduce their consumption. I chose an optimistic value of 90% to consider the survey completely representative of the 7% who liked the page, and then a bit more for those who reduced their consumption but did not like the page. My pessimistic value was 95%, assuming everyone who did not like the survey went unchanged and assuming a small response bias among those who liked the page but chose not to take the survey.

For the pamphlets, however, there should be no response bias since the entire population of college students was surveyed from randomly, and no one was said to reject taking the survey.

Additional People Are Being Reached

In the Facebook survey, those who said they reduced their meat consumption were also asked if they influenced any of their friends and family to also reduce eating meat, and found that they usually produced 0.86 additional reducers.

This figure seems very high, but I do strongly expect the figure to be positive -- people who reduce eating meat will talk about it sometimes, essentially becoming free advertisements. I'd be very surprised if they ended up being a net negative.

Accounting for Product Elasticity

Another way to boost the effectiveness of the estimate is to be more accurate about what happens when someone stops eating meat. The change isn't from the actual refusal to eat, but rather from the reduced demand for meat, which leads to a reduced supply. Following the laws of economics, however, this reduction won't necessarially be one-for-one, but rather depend on the elasticity of product demand and supply. By getting this number, we can find out how much meat is reduced for every meat not demanded.

My guesses in the calculator come from the following sources, some of which are PDFs: Beef #1, Beef #2, Dairy #1, Dairy #2, Pork #1, Pork #2, Egg #1, Egg #2, Poultry, Salmon, and for all fish.

Putting It All Together

Implementing the formula on the calculator, we end up with an estimate of $0.03 to $36.52 to reduce one year of suffering on a factory farm based on the Facebook ad data and an estimate of $0.02 to $65.92 based on the pamphlet data.

Of course, many people are skeptical of these figures. Perhaps surprisingly, so am I. I'm trying to strike a balance between being an advocate of vegan outreach as a very promising path for making the world a better place, while not losing sight of the methodological hurdles that have not yet been met, and open to the possibility that I'm wrong about this.

The big methodological elephant in the room is that my entire cost estimate depends on having a plausible guess for how likely someone is to change their behavior based on seeing an advertisement.

I feel slightly reassured because:

There are two surveys for two different media, and they both provide estimates of impact that agree with each other.
These estimates also match anecdotes from leafleters about approximately how many people come back and say they went vegetarian because of a pamphlet.
Even if we were to take the simple calculator and drop the "2% chance of getting four years of vegetarianism" assumption down to, say, a pessimistic "0.1% chance of getting one year" conversion rate, the estimate is still not too bad -- $0.91 to avert a year of suffering.
More studies are on the way. Nick Cooney is going to do a bunch more to study leaflets, and Xio Kikauka and Joey Savoie have publicly published some survey methodology [Google Docs].

That said, the possibility for desirability bias in the survey is a large concern as long as the surveys continue to be from overt animal welfare groups and continue to clearly state that they're looking for reductions in meat consumption.

Also, so long as surveys are only given to people that remember the leaflet or advertisement, there will be a strong possibility of response bias, as those who remember the ad are more likely to be the ones who changed their behavior. We can attempt to compensate for these things, but we can only do so much.

Furthermore, and more worrying, there's a concern that the surveys are just measuring normal drift in vegetarianism, without any changes being attributable to the ads themselves. For example, imagine that every year, 2% of people become vegetarians and 2% quit. Surveying these people at random and not capturing those who quit will end up finding a 2% conversion rate.

How can we address these? I think all three problems can be solved with a decent control group, whether it be a group of people that receive a leaflet not about vegetarianism, or no leaflet at all. Luckily, Kikauka and Savoie's survey intend to do just that.

Jeff Kaufman has a good proposal for a survey design I'd like to see implemented in this area.

Market Saturation and Diminishing Marginal Returns?

Another concern is that there are diminishing marginal returns to these ads. As the critique goes, there are only so many people that will be easily swayed by the advertisement, and once all of them are quickly reached by Facebook ads and pamphlets, things will dry up.

Unlike the others, I don't think this criticism works well. After all, even if it were true, it still would be worthwhile to take the market as far as it will go, and we can keep monitoring for saturation and find the point where it's no longer cost-effective.

However, I don't think the market has been tapped up yet at all. According to Nick Cooney [PDF], there are still many opportunities in foreign markets and outside the young, college kid demographic.

The Conjunction Fallacy?

The conjunction fallacy is a classic fallacy that reminds us that no matter what, the chance of event A happening can never be smaller than the chance of event A happening, followed by event B. For example, the probability that Linda is a bank teller will always be larger than (or equal to) the probability that Linda is a bank teller and a feminist.

What does this mean for vegetarian outreach? Well, for the simple calculator, we're estimating five factors. In the complex calculator, we're estimating 90 factors. Even if each factor is 99% likely to be correct, the chance that all five are right is 95%, and the chance that all 50 are right is only 60%. If each factor is only 90% likely to be correct, the complex calculator will be right with a probability of 0.5%!

This is a cause for concern, but I don't think there's any way around this. It's just an inherent problem with estimation. Hopefully we'll be balanced by (1) using the different bounds and (2) hoping underestimates and overestimates will cancel each other out.

Conversion and The 100 Yard Line

Something we should take into account that helps the case for this outreach rather than hurts it is the idea that conversions aren't binary -- someone can be pushed by the ad to be more likely to reduce their meat intake as opposed to fully converted. As Brian Tomasik puts it:

Yes, some of the people we convince were already on the border, but there might be lots of other people who get pushed further along and don’t get all the way to vegism by our influence. If we picture the path to vegism as a 100-yard line, then maybe we push everyone along by 20 yards. 1/5 of people cross the line, and this is what we see, but the other 4/5 get pushed closer too. (Obviously an overly simplistic model, but it illustrates the idea.)

This would be either very difficult or outright impossible to capture in a survey, but is something to take into account.

Three Places I Might Donate Before Donating to Vegan Outreach

When all is said and done, I like the case for funding this outreach. However, I think there are three other possibilities along these lines that I find more promising:

Funding the research of vegan outreach: There needs to be more and higher-quality studies of this before one can feel confident enough in the cost-effectiveness of this outreach. However, initial results are very promising, and the value of information of more studies is therefore very high. Studies can also find ways to advertise more effectively, increasing the impact of each dollar spent. Right now, however, it looks like all ongoing studies are fully funded, but if there were opportunities to fund more, I would jump on it.

Funding Effective Animal Activism: EAA is an organization pushing for more cost-effectiveness in the domain of nonhuman animal welfare and is working to further evaluate what opportunities are the best, Givewell-style. Giving them more money can potentially attract a lot more attention to this outreach, and get it more scrutiny, research, and money down the line.

Funding Centre for Effective Altruism: Overall, it might just be better to get more people involved in the idea of giving effectively, and then getting them interested in vegan outreach, among other things.

Conclusion

Vegan outreach is a promising, though not fully studied, method of outreach that deserves both excitement and skepticism. Should one put money into it? Overall, I'd take a guarded approach of putting in just enough money to help the organizations learn, develop better cost-effective measurements and transparency, and become more effective. It shouldn't be too long before this area will become studied well enough to have good confidence in how things are doing.

More studies should be developed that explore advertising vegetarianism in a wide variety of media in a wide variety of ways, with decent control groups.

I look forward to seeing how this develops. Don't forget to play around with my calculator.

Footnotes

[1]: Cost effectiveness in years of suffering prevented per dollar = (Pamphlets / dollar) * (Conversions / pamphlet) * (Veg years / conversion) * (Animals saved / veg year) * (Years lived / animal).

Plugging in 80K's values... Cost effectiveness = (Pamphlets / dollar) * 0.01 to 0.03 * 25 * 100 * (Years lived / animal)

Filling in the gaps with my best guesses... Cost effectiveness = 5 * 0.01 to 0.03 * 25 * 100 * 0.90 = 112.5 to 337.5 years of suffering averted per dollar
I personally think 25 veg-years per conversion on average is possible but too high; I personally err from 4 to 7.

[2]: I feel like there's an error in this calculation or that Kaufman might disagree with my assumptions of number of animals or days per animal, because I've been told before that these estimates with this method are supposed to be about an order of magnitude higher than other estimates. However, I emailed Kaufman and he seemed to not find any fault with the calculation, though he does think the methodology is bad and the calculation should not be taken at face value.

[3]: I calculated the number of vegetarians by eyeballing about how many people said they no longer eat fish, which I'd guess only a vegetarian would be willing to give up.

[4]: 32 vegetarians / 104 people = 30.7%. That population is 8.5% (7% for likes + 1.5% for the starter kit) of the overall population, leading to 2.61% (30.7% * 8.5%).

[5]: Formula is [(Number Meat Chickens)(Days Alive) + (Number Egg Chickens)(Days Alive) + (Number Beef Cows)(Days Alive) + (Number Milk Cows)(Days Alive) + (Number Fish)(Days Alive)] / (Total Number Animals). ...Plugging things in: [(28)(42) + (2)(365) + (0.125)(365) + (0.033)(1460) + (225)(365)] / 255.16] = 329.6 days

[6]: Cost effectiveness in amount of days prevented per dollar = (People Reached / Dollar + (People Reached / Dollar * Additional People Reached / Direct Reach * Response Bias * Desirability Bias)) * Years Spent Reducing * (((Percent Increasing Beef * Increase Value) + (Percent Staying Same with Beef * Staying Same Value) + (Percent Decreasing Beef Slightly * Decrease Slightly Value) + (Percent Decreasing Beef Significantly * Decrease Significantly Value) + (Percent Eliminating Beef * Elimination Value) + (Percent Never Ate Beef * Never Ate Value)) * Normal Beef Consumption * Beef Elasticity * (Average Beef Lifespan + Days of Suffering from Beef Slaughter)) + (((Percent Increasing Dairy * Increase Value) + (Percent Staying Same with Dairy * Staying Same Value) + (Percent Decreasing Dairy Slightly * Decrease Slightly Value) + (Percent Decreasing Dairy Significantly * Decrease Significantly Value) + (Percent Eliminating Dairy * Elimination Value) + (Percent Never Ate Dairy * Never Ate Value)) * Normal Dairy Consumption * Dairy Elasticity * (Average Dairy Lifespan + Days of Suffering from Dairy Slaughter)) + (((Percent Increasing Pig * Increase Value) + (Percent Staying Same with Pig * Staying Same Value) + (Percent Decreasing Pig Slightly * Decrease Slightly Value) + (Percent Decreasing Pig Significantly * Decrease Significantly Value) + (Percent Eliminating Pig * Elimination Value) + (Percent Never Ate Pig * Never Ate Value)) * Normal Pig Consumption * Pig Elasticity * (Average Pig Lifespan + Days of Suffering from Pig Slaughter)) + (((Percent Increasing Broiler Chicken * Increase Value) + (Percent Staying Same with Broiler Chicken * Staying Same Value) + (Percent Decreasing Broiler Chicken Slightly * Decrease Slightly Value) + (Percent Decreasing Broiler Chicken Significantly * Decrease Significantly Value) + (Percent Eliminating Broiler Chicken * Elimination Value) + (Percent Never Ate Broiler Chicken * Never Ate Value)) * Normal Broiler Chicken Consumption * Broiler Chicken Elasticity * (Average Broiler Chicken Lifespan + Days of Suffering from Broiler Chicken Slaughter)) + (((Percent Increasing Egg * Increase Value) + (Percent Staying Same with Egg * Staying Same Value) + (Percent Decreasing Egg Slightly * Decrease Slightly Value) + (Percent Decreasing Egg Significantly * Decrease Significantly Value) + (Percent Eliminating Egg * Elimination Value) + (Percent Never Ate Egg * Never Ate Value)) * Normal Egg Consumption * Egg Elasticity * (Average Egg Lifespan + Days of Suffering from Egg Slaughter)) + (((Percent Increasing Turkey * Increase Value) + (Percent Staying Same with Turkey * Staying Same Value) + (Percent Decreasing Turkey Slightly * Decrease Slightly Value) + (Percent Decreasing Turkey Significantly * Decrease Significantly Value) + (Percent Eliminating Turkey * Elimination Value) + (Percent Never Ate Turkey * Never Ate Value)) * Normal Turkey Consumption * Turkey Elasticity * (Average Turkey Lifespan + Days of Suffering from Turkey Slaughter)) + (((Percent Increasing Farmed Fish * Increase Value) + (Percent Staying Same with Farmed Fish * Staying Same Value) + (Percent Decreasing Farmed Fish Slightly * Decrease Slightly Value) + (Percent Decreasing Farmed Fish Significantly * Decrease Significantly Value) + (Percent Eliminating Farmed Fish * Elimination Value) + (Percent Never Ate Farmed Fish * Never Ate Value)) * Normal Farmed Fish Consumption * Farmed Fish Elasticity * (Average Farmed Fish Lifespan + Days of Suffering from Farmed Fish Slaughter)) + (((Percent Increasing Sea Fish * Increase Value) + (Percent Staying Same with Sea Fish * Staying Same Value) + (Percent Decreasing Sea Fish Slightly * Decrease Slightly Value) + (Percent Decreasing Sea Fish Significantly * Decrease Significantly Value) + (Percent Eliminating Sea Fish * Elimination Value) + (Percent Never Ate Sea Fish * Never Ate Value)) * Normal Sea Fish Consumption * Sea Fish Elasticity * Days of Suffering from Sea Fish Slaughter) * Response Bias * Desirability Bias

[7]: Feel free to check the formula for accuracy and also check to make sure the calculator implements the formula correctly. I worry that the added accuracy from the complex calculator is outweighed by the risk that the formula is wrong.

Edited 18 June to correct two typos and update footnote #2.

Also cross-posted on my blog.

466 comments

A personal history of involvement with effective altruism

Tue, 11 Jun 2013 04:49:45 +0000

Submitted by JonahSinick • 6 votes • 17 comments

Over the coming weeks, I intend to write up a history of the different parts of the effective altruist movement and their interrelations. It’s natural to start with the part that I know best: the history of my own involvement with effective altruism.

Interest in altruism rooted in literature

My interest in altruism traces to early childhood. Unbeknownst to me, my verbal comprehension ability was unusually high relative to my other cognitive abilities, and for this reason, I gravitated strongly toward reading. Starting from the age of six, I spent hours a day reading fiction. I found many of the stories that I read to be emotionally compelling, and identified with the characters.

My interest is altruism is largely literary in origin — I perceive the sweep of history to be a story, and I want things to go well for the characters, and want the story to have a happy ending. I was influenced both by portrayals of sympathetic, poor characters in need, and by stories of the triumph of the human spirit, and I wanted to help the downtrodden, and contribute to the formation of peak positive human experiences.

I sometimes wonder whether there are other people with altruistic tendencies that are literary in origin, and whether they would be good candidate members of the effective altruist movement. There is some history of artists having altruistic goals. The great painter Vincent van Gogh moved to an impoverished coal mine to preach and minister to the sick. The great mathematician Alexander Grothendieck gave shelter to the homeless.

An analytical bent, and utilitarianism

When I was young, I had vague and dreamy hopes about how I might make the world a better place. As I grew older, I found myself more focused on careful reasoning and rationality.

In high school, I met Dario Amodei, who introduced me to utilitarianism. The ethical framework immediately resonated with me. For me, it corresponded to valuing the well being of all characters in the story — a manifestation of universal love.

This was the birth of my interest in maximizing aggregated global welfare. Maximizing aggregated global welfare corresponds to maximizing cost-effectiveness, and so this can be thought of as the origin of my interest in effective altruism.

I believe that I would have developed interest in global welfare, and in effective altruism, on my own accord, without encountering any members of the effective altruist movement. But for reasons that I describe below, if not for meeting these people, I don’t think that my interests would have been actionable.

Epistemic paralysis

My analytical bent had a downside.

Issues pertaining to the human world are very complex, and there aren’t clear-cut objective answers to the question of how best to make the world a better place. On a given issue, there are many arguments for a given position, and many counterarguments to the arguments, and many counterarguments to the counterarguments, and so on.

Contemplating these resulted in my falling into a state of epistemic learned helplessness. I became convinced that it's not possible to rationally develop confidence in views concerning how to make the world a better place.

Enter GiveWell

In 2007, my college friend Brian Tomasik pointed me to GiveWell. At the time, GiveWell had just launched, and there wasn’t very much on the website, so I soon forgot about it.

In 2009, my high school friend Dario, who had introduced me to utilitarianism, pointed me to GiveWell again. By this point, there was much more information available on the GiveWell website.

I began following GiveWell closely. I was very impressed by the fact that co-founders Holden and Elie seemed to be making sense of the ambiguous world of effective philanthropy. I hadn’t thought that it was possible to reason so well about the human world. This made effective altruism more credible in my eyes, and inspired me. If hadn’t encountered GiveWell, I may not have gotten involved with the effective philanthropy movement at all, although I may have become involved through interactions of Less Wrong, and I may have gone on to do socially valuable work in math education.

I became progressively more impressed by GiveWell over time, and wanted to become involved. In 2011, I did volunteer work for GiveWell, and in 2012, I began working at GiveWell as a research analyst.

While working at GiveWell, I learned a great deal about how to think about philanthropy, and about epistemology more generally. A crucial development in my thinking was the gradual realization that:

Cost-effectiveness estimates that are supported by a single line of evidence are unreliable.
Investigation generally reveals that interventions that appear highly cost-effective relative to other interventions are often much worse (in a relative sense) than one would initially guess.
One can become much more confident in one’s assessment of the value of a philanthropic intervention by examining it from many different angles.

I wrote about this realization in my post Robustness of Cost-Effectiveness Estimates and Philanthropy.

This shift in my thinking gradually percolated, and I realized that my entire epistemological framework had been seriously flawed, because I was relying too much on a small number of relatively strong arguments rather than a large number of independent weak arguments.

Many people had tried to explain this to me in the past, but I was unable to understand what they were driving at, and it was only through my work at GiveWell and my interactions with my coworkers that I was finally able to understand. The benefits of this realization have spanned many aspects of my life, and have substantially increased my altruistic human capital.

If GiveWell hadn’t existed, it’s very possible that I wouldn’t have learned these things. If Dario hadn’t pointed me to GiveWell, I’m sure that I would have encountered GiveWell eventually, but it may have been too late for it to be possible for me to work there, and so I may not have had the associated learning opportunities.

My involvement with GiveWell also facilitated my meeting Vipul Naik, the founder of Open Borders. We’ve had many fruitful interactions related to maximizing global welfare, and if I hadn’t met him through GiveWell, it may have been years before we met.

The significance of Less Wrong

Several people pointed me to Overcoming Bias and Less Wrong starting in 2008, but at the time the posts didn’t draw me in relative to the fascination of reciprocity laws in algebraic number theory. In early 2010, Brian Tomasik pointed me to some of Yvain’s articles on Less Wrong. With the background context of me following GiveWell, Yvain’s posts on utilitarianism really resonated with me. So I started reading Less Wrong.

I met many impressive people who are seriously interested in effective altruism through Less Wrong. Among these are:

Nick Beckstead — A new postdoc at Oxford University (specifically, the Future of Humanity Institute) who wrote the thesis On the Overwhelming Importance of Shaping the Far Future.
Paul Christiano and Jacob Steinhardt — graduate students in theory of computing and machine learning at Berkeley and Stanford (respectively). Jacob is a Hertz Fellow, and Paul coauthored a 48 page paper titled Quantum Money from Hidden Subspaces as an undergraduate. Paul has thought a great deal about rational altruism, and Jacob is a member of The Vannevar Group, which aims to accelerate scientific progress to create the greatest amount of social good.
Qiaochu Yuan — A math graduate student at Berkeley who has the sixth highest karma score on MathOverflow. I’ve been enjoying learning math from Qiaochu.
Carl Shulman and Dan Keys. I’ve found very intellectually stimulating and have learned a great deal from them.
Luke Muehlhauser — The executive director of MIRI.
Others — Including Julia Galef, Anna Salamon, Katja Grace, Louie Helm.

They’ve helped me retain my motivation to do the most good, and have aided me in thinking about effective altruism. They constitute a substantial chunk of the most impressive people who I know in my age group.

It’s genuinely unclear whether I would have gotten to know these people if Eliezer hadn’t started Less Wrong.

Closing summary

My innate inclinations got me interested in effective altruism, but they probably wouldn’t have sufficed for my interest to be actionable. Beyond my innate inclinations, the things that stand out most in my mind as having been crucial are

Dario introducing me to GiveWell
Working at GiveWell
Meeting Vipul through GiveWell
Eliezer starting Less Wrong
Meeting peers through Less Wrong

Working at GiveWell substantially increased my altruistic human capital. I’ve learned a great deal from the GiveWell staff, from Vipul, and from the members of the Less Wrong community listed above. We’ve had fruitful collaborations, and they’ve helped me retain my motivation to do the most good.

The personal growth benefits that I derived from working at GiveWell are unusual, if only because GiveWell’s staff is small. The networking benefits from Less Wrong are shared by many others.

Note: I formerly worked as a research analyst at GiveWell. All views here are my own.

This post is cross-posted at www.effective-altruism.com.

17 comments

Anticipating critical transitions

Sun, 09 Jun 2013 16:28:51 +0000

Submitted by PhilGoetz • 16 votes • 51 comments

(Mathematicians may find this post painfully obvious.)

I read an interesting puzzle on Stephen Landsburg's blog that generated a lot of disagreement. Stephen offered to bet anyone $15,000 that the average results of a computer simulation, run 1 million times, would be close to his solution's prediction of the expected value.

Landsburg's solution is in fact correct. But the problem involves a probabilistic infinite series, a kind used often on less wrong in a context where one is offered some utility every time one flips a coin and it comes up heads, but loses everything if it ever comes up tails. Landsburg didn't justify the claim that a simulation could indicate the true expected outcome of this particular problem. Can we find similar-looking problems for which simulations give the wrong answer? Yes.

Here's Perl code to estimate by simulation the expected value of the series of terms 2^k / k from k = 1 to infinity, with a 50% chance of stopping after each term.

my $bigsum = 0;
for (my $trial = 0; $trial < 1000000; $trial++) {
    my $sum = 0;
    my $top = 2;
    my $denom = 1;
    do {
        $sum += $top / $denom;
        $top *= 2;
        $denom += 1;
    }
    while (rand(1) < .5);
    $bigsum += $sum;
}
my $ave = $bigsum / $runs;
print "ave sum=$ave\n";

(If anyone knows how to enter a code block on this site, let me know. I used the "pre" tag, but the site stripped out my spaces anyway.)

Running it 5 times, we get the answers

ave sum=7.6035709716983

ave sum=8.47543819631431

ave sum=7.2618950097739

ave sum=8.26159741956747

ave sum=7.75774577340324

So the expected value is somewhere around 8?

No; the expected value is given by the sum of the harmonic series, which diverges, so it's infinite. Later terms in the series are exponentially larger, but exponentially less likely to appear.

Some of you are saying, "Of course the expected value of a divergent series can't be computed by simulation! Give me back my minute!" But many things we might simulate with computers, like the weather, the economy, or existential risk, are full of power law distributions that might not have a convergent expected value. People have observed before that this can cause problems for simulations (see The Black Swan). What I find interesting is that the output of the program above doesn't look like something inside it diverges. It looks almost normal. So you could run your simulation many times and believe that you had a grip on its expected outcome, yet be completely mistaken.

In real-life simulations (that sounds wrong, doesn't it?), there's often some system property that drifts slowly, and some critical value of that system property above which some distribution within the simulation diverges. Moving above that critical value doesn't suddenly change the output of the simulation in a way that gives an obvious warning. But the expected value of keeping that property below that critical value in the real-life system being simulated can be very high (or even infinite), with very little cost.

Is there a way to look at a simulation's outputs, and guess whether a particular property is near some such critical threshold? Better yet, is there a way to guess whether there exists some property in the system nearing some such threshold, even if you don't know what it is?

The October 19, 2012 issue of Science contains an article on just that question: "Anticipating critical transitions", Marten Scheffer et al., p. 344. It reviews 28 papers on systems and simulations, and lists about a dozen mathematical approaches used to estimate nearness to a critical point. These include:

Critical slowing down: When the system is near a critical threshold, it recovers slowly from small perturbations. One measure of this is autocorrelation at lag 1, meaning the correlation between the system's output at times T and T-1. Counterintuitively, a higher autocorrelation at lag one by itself suggests that the system is more predictable than before, but may actually indicate it is less predictable. The more predictable system reverts to its mean; the unpredictable system has no mean.
Flicker: Instead of having a single stable state that the system reverts to after perturbation, an additional stable state appears, and the system flickers back and forth between the two states.
Dominant eigenvalue: I haven't read the paper that explains what this paper means when it cites this, but I do know that you can predict when a helicopter engine is going to malfunction by putting many sensors on it, running PCA on time-series data for those sensors to get a matrix that projects their output into just a few dimensions, then reading their output continuously and predicting failure anytime the PCA-projected output vector moves a lot. That probably is what they mean.

So if you're modeling global warming, running your simulation a dozen times and averaging the results may be misleading. [1] Global temperature has sudden [2] dramatic transitions, and an exceptionally large and sudden one (15C in one million years) neatly spans the Earth's greatest extinction event so far on the Permian-Triassic boundary [3]. It's more important to figure out what the critical parameter is and where its critical point is than to try and estimate how many years it will be before Manhattan is underwater. The "expected rise in water level per year" may not be easily-answerable by simulation [4].

And if you're thinking about betting Stephen Landsburg $15,000 on the outcome of a simulation, make sure his series converges first. [5]

[1] Not that I'm particularly worried about global warming.

[2] Geologically sudden.

[3] Sun et al., "Lethally hot temperatures during the early Triassic greenhouse", Science 338 (Oct. 19 2012) p.366, see p. 368. Having just pointed out that an increase of .000015C/yr counts as a "sudden" global warming event, I feel obligated to also point out that the current increase is about .02C/yr.

[4] It will be answerable by simulation, since rise in water level can't be infinite. But you may need a lot more simulations than you think.

[5] Better yet, don't bet against Stephen Landsburg.

51 comments

Robust Cooperation in the Prisoner's Dilemma

Fri, 07 Jun 2013 08:30:25 +0000

Submitted by orthonormal • 61 votes • 108 comments

I'm proud to announce the preprint of Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic, a joint paper with Patrick LaVictoire (me), Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, and Eliezer Yudkowsky.

This paper was one of three projects to come out of the 2nd MIRI Workshop on Probability and Reflection in April 2013, and had its genesis in ideas about formalizations of decision theory that have appeared on LessWrong. (At the end of this post, I'll include links for further reading.)

Below, I'll briefly outline the problem we considered, the results we proved, and the (many) open questions that remain. Thanks in advance for your thoughts and suggestions!

Background: Writing programs to play the PD with source code swap

(If you're not familiar with the Prisoner's Dilemma, see here.)

The paper concerns the following setup, which has come up in academic research on game theory: say that you have the chance to write a computer program X, which takes in one input and returns either Cooperate or Defect. This program will face off against some other computer program Y, but with a twist: X will receive the source code of Y as input, and Y will receive the source code of X as input. And you will be given your program's winnings, so you should think carefully about what sort of program you'd write!

Of course, you could simply write a program that defects regardless of its input; we call this program DefectBot, and call the program that cooperates on all inputs CooperateBot. But with the wealth of information afforded by the setup, you might wonder if there's some program that might be able to achieve mutual cooperation in situations where DefectBot achieves mutual defection, without thereby risking a sucker's payoff. (Douglas Hofstadter would call this a perfect opportunity for superrationality...)

Previously known: CliqueBot and FairBot

And indeed, there's a way to do this that's been known since at least the 1980s. You can write a computer program that knows its own source code, compares it to the input, and returns C if and only if the two are identical (and D otherwise). Thus it achieves mutual cooperation in one important case where it intuitively ought to: when playing against itself! We call this program CliqueBot, since it cooperates only with the "clique" of agents identical to itself.

There's one particularly irksome issue with CliqueBot, and that's the fragility of its cooperation. If two people write functionally analogous but syntactically different versions of it, those programs will defect against one another! This problem can be patched somewhat, but not fully fixed. Moreover, mutual cooperation might be the best strategy against some agents that are not even functionally identical, and extending this approach requires you to explicitly delineate the list of programs that you're willing to cooperate with. Is there a more flexible and robust kind of program you could write instead?

As it turns out, there is: in a 2010 post on LessWrong, cousin_it introduced an algorithm that we now call FairBot. Given the source code of Y, FairBot searches for a proof (of less than some large fixed length) that Y returns C when given the source code of FairBot, and then returns C if and only if it discovers such a proof (otherwise it returns D). Clearly, if our proof system is consistent, FairBot only cooperates when that cooperation will be mutual. But the really fascinating thing is what happens when you play two versions of FairBot against each other. Intuitively, it seems that either mutual cooperation or mutual defection would be stable outcomes, but it turns out that if their limits on proof lengths are sufficiently high, they will achieve mutual cooperation!

The proof that they mutually cooperate follows from a bounded version of Löb's Theorem from mathematical logic. (If you're not familiar with this result, you might enjoy Eliezer's Cartoon Guide to Löb's Theorem, which is a correct formal proof written in much more intuitive notation.) Essentially, the asymmetry comes from the fact that both programs are searching for the same outcome, so that a short proof that one of them cooperates leads to a short proof that the other cooperates, and vice versa. (The opposite is not true, because the formal system can't know it won't find a contradiction. This is a subtle but essential feature of mathematical logic!)

Generalization: Modal Agents

Unfortunately, FairBot isn't what I'd consider an ideal program to write: it happily cooperates with CooperateBot, when it could do better by defecting. This is problematic because in real life, the world isn't separated into agents and non-agents, and any natural phenomenon that doesn't predict your actions can be thought of as a CooperateBot (or a DefectBot). You don't want your agent to be making concessions to rocks that happened not to fall on them. (There's an important caveat: some things have utility functions that you care about, but don't have sufficient ability to predicate their actions on yours. In that case, though, it wouldn't be a true Prisoner's Dilemma if your values actually prefer the outcome (C,C) to (D,C).)

However, FairBot belongs to a promising class of algorithms: those that decide on their action by looking for short proofs of logical statements that concern their opponent's actions. In fact, there's a really convenient mathematical structure that's analogous to the class of such algorithms: the modal logic of provability (known as GL, for Gödel-Löb).

So that's the subject of this preprint: what can we achieve in decision theory by considering agents defined by formulas of provability logic?

More formally (skip the next two paragraphs if you're willing to trust me), we inductively define the class of "modal agents" as formulas using propositional variables and logical connectives and the modal operator (which represents provability in some base-level formal system like Peano Arithmetic), of the form , where is fully modalized (i.e. all instances of variables are contained in an expression ), and with each corresponding to a fixed modal agent of lower rank. For example, FairBot is represented by the modal formula .

When two modal agents play against each other, the outcome is given by the unique fixed point of the system of modal statements, where the variables are identified with each other so that represents the expression , represents , and the represent the actions of lower-rank modal agents against and vice-versa. (Modal rank is defined as a natural number, so this always bottoms out in a finite number of modal statements; also, we interpret outcomes as statements of provability in Peano Arithmetic, evaluated in the model where PA is consistent, PA+Con(PA) is consistent, and so on. See the paper for the actual details.)

The nice part about modal agents is that there are simple tools for finding the fixed points without having to search through proofs; in fact, Mihaly and Marcello wrote up a computer program to deduce the outcome of the source-code-swap Prisoner's Dilemma between any two (reasonably simple) modal agents. These tools also made it much easier to prove general theorems about such agents.

PrudentBot: The best of both worlds?

Can we find a modal agent that seems to improve on FairBot? In particular, we should want at least the following properties:

It should be un-exploitable: if our axioms are consistent in the first place, then it had better only end up cooperating when it's mutual.
It should cooperate with itself, and also mutually cooperate with FairBot (both are, common-sensically, the best actions in those cases).
It should defect, however, against CooperateBot and lots of similarly exploitable modal agents.

It's nontrivial that such an agent exists: you may remember the post I wrote about the Masquerade agent, which is a modal agent that does almost all of those things (it doesn't cooperate with the original FairBot, though it does cooperate with some more complicated variants), and indeed we didn't find anything better until after we had Mihaly and Marcello's modal-agent-evaluator to help us.

But as it turns out, there is such an agent, and it's pretty elegant: we call it PrudentBot, and its modal version cooperates with another agent Y if and only if (there's a proof in Peano Arithmetic that Y cooperates with PrudentBot and there's a proof in PA+Con(PA) that Y defects against DefectBot). This agent can be seen to satisfy all of our criteria. But is it optimal among modal agents, by any reasonable criterion?

Results: Obstacles to Optimality

It turns out that, even within the class of modal agents, it's hard to formulate a definition of optimality that's actually true of something, and which meaningfully corresponds to our intuitions about the "right" decisions on decision-theoretic problems. (This intuition is not formally defined, so I'm using scare quotes.)

There are agents that give preferential treatment to DefectBot, FairBot, or even CooperateBot, compared to PrudentBot, though these agents are not ones you'd program in an attempt to win at the Prisoner's Dilemma. (For instance, one agent that rewards CooperateBot over PrudentBot is the agent that cooperates with Y iff PA proves that Y cooperates against DefectBot; we've taken to jokingly calling that agent TrollBot.) One might well suppose that a modal agent could still be optimal in the sense of making the "right" decision in every case, regardless of whether it's being punished for some other decision. However, this is not the only obstacle to a useful concept of optimality.

The second obstacle is that any modal agent only checks proofs at some finite number of levels on the hierarchy of formal systems, and agents that appear indistinguishable at all those levels may have obviously different "right" decisions. And thirdly, an agent might mimic another agent in such a way that the "right" decision is to treat the mimic differently from the agent it imitates, but in some cases one can prove that no modal agent can treat the two differently.

These three strikes appear to indicate that if we're looking to formalize more advanced decision theories, modal agents are too restrictive of a class to work with. We might instead allow things like quantifiers over agents, which would invalidate these specific obstacles, but may well introduce new ones (and certainly would make for more complicated proofs). But for a "good enough" algorithm on the original problem (assuming that the computer will have lots of computational resources), one could definitely do worse than submit a finite version of PrudentBot.

Why is this awesome, and what's next?

In my opinion, the result of Löbian cooperation deserves to be published for its illustration of Hofstadterian superrationality in action, apart from anything else! It's really cool that two agents reasoning about each other can in theory come to mutual cooperation for genuine reasons that don't have to involve being clones of each other (or other anthropic dodges). It's a far cry from a practical approach, of course, but it's a start: mathematicians always begin with a simplified and artificial model to see what happens, then add complications one at a time.

As for what's next: First, we don't actually know that there's no meaningful non-vacuous concept of optimality for modal agents; it would be nice to know that one way or another. Secondly, we'd like to see if some other class of agents contains a simple example with really nice properties (the way that classical game theory doesn't always have a pure Nash equilibrium, but always has a mixed one). Thirdly, we might hope that there's an actual implementation of a decision theory (TDT, UDT, etc) in the context of program equilibrium.

If we succeed in the positive direction on any of those, we'd next want to extend them in several important ways: using probabilistic information rather than certainty, considering more general games than the Prisoner's Dilemma (bargaining games have many further challenges, and games of more than two players could be more convoluted still), etc. I personally hope to work on such topics in future MIRI workshops.

Prisoner's Dilemma (with visible source code) Tournament

Fri, 07 Jun 2013 08:30:20 +0000

Submitted by AlexMennen • 45 votes • 168 comments

After the iterated prisoner's dilemma tournament organized by prase two years ago, there was discussion of running tournaments for several variants, including one in which two players submit programs, each of which are given the source code of the other player's program, and outputs either “cooperate” or “defect”. However, as far as I know, no such tournament has been run until now.

Here's how it's going to work: Each player will submit a file containing a single Scheme lambda-function. The function should take one input. Your program will play exactly one round against each other program submitted (not including itself). In each round, two programs will be run, each given the source code of the other as input, and will be expected to return either of the symbols “C” or “D” (for "cooperate" and "defect", respectively). The programs will receive points based on the following payoff matrix:

“Other” includes any result other than returning “C” or “D”, including failing to terminate, throwing an exception, and even returning the string “Cooperate”. Notice that “Other” results in a worst-of-both-worlds scenario where you get the same payoff as you would have if you cooperated, but the other player gets the same payoff as if you had defected. This is an attempt to ensure that no one ever has incentive for their program to fail to run properly, or to trick another program into doing so.

Your score is the sum of the number of points you earn in each round. The player with the highest score wins the tournament. Edit: There is a 0.5 bitcoin prize being offered for the winner. Thanks, VincentYu!

Details:
All submissions must be emailed to wardenPD@gmail.com by July 5, at noon PDT. Your email should also say how you would like to be identified when I announce the tournament results.
Each program will be allowed to run for 10 seconds. If it has not returned either “C” or “D” by then, it will be stopped, and treated as returning “Other”. For consistency, I will have Scheme collect garbage right before each run.
One submission per person or team. No person may contribute to more than one entry. Edit: This also means no copying from each others' source code. Describing the behavior of your program to others is okay.
I will be running the submissions in Racket. You may be interested in how Racket handles time (especially the (current-milliseconds) function), threads (in particular, “thread”, “kill-thread”, “sleep”, and “thread-dead?”), and possibly randomness.
Don't try to open the file you wrote your program in (or any other file, for that matter). I'll add code to the file before running it, so if you want your program to use a copy of your source code, you will need to use a quine. Edit: No I/O of any sort.
Unless you tell me otherwise, I assume I have permission to publish your code after the contest.
You are encouraged to discuss strategies for achieving mutual cooperation in the comments thread.
I'm hoping to get as many entries as possible. If you know someone who might be interested in this, please tell them.
It's possible that I've said something stupid that I'll have to change or clarify, so you might want to come back to this page again occasionally to look for changes to the rules. Any edits will be bolded, and I'll try not to change anything too drastically, or make any edits late in the contest.

Here is an example of a correct entry, which cooperates with you if and only if you would cooperate with a program that always cooperates (actually, if and only if you would cooperate with one particular program that always cooperates):

(lambda (x)
(if (eq? ((eval x) '(lambda (y) 'C)) 'C)
'C
'D))

168 comments

Model Stability in Intervention Assessment

Thu, 06 Jun 2013 23:24:24 +0000

Submitted by Jonathan_Lee • 5 votes • 5 comments

In this post, I hope to examine the Bayesian Adjustment paradigm presented by Holden Karnofsky of Givewell from a mathematical viewpoint, in particular looking at how we can rigorously manage the notion of uncertainty in our models and the stability of an estimate. Several recent posts have touched on related issues.

In practise, we will need to have some substantive prior on the likely range of impacts that interventions can achieve, and I will look briefly at what kinds of log-ranges are supported in the literature, and the extent to which these can preclude extreme impact scenarios. I will then briefly look at less formal notions of confidence in a model, which may be more tractable either computationally or for heuristic purposes than a formal bayesian approach.

Bayesian Adjustment, and the A_p distribution

In the setting originally proposed, the BA framework takes a background prior on impacts and a noisy measurement of fixed variance of a fixed impact parameter. In this setting, the BA approach is provably correct. Unfortunately, the real world is not so accommodating; for general evidence about an intervention, the BA approach is not fully Bayesian. In this sense it unavoidably miscounts evidence. The general problem can be illustrated by working through the process formally. Consider propositions:

x := Has Impact x,
E := Background data,
C := there exists a given computation or argument to a given impact y.

We suppose for the framework that we have P(x|E), P(x|C) for each x. Since the set of propositions {x} are disjoint and exhaustive, these form distributions. For inference, what we actually want is P(x|EC). In the BA framework, we compute P(x|E)P(x|C) for each x, and normalise to get a distribution. Computing a bayesian update, we have:

P(x|EC) = P(xEC)/P(EC) = P(C|xE)P(x|E)/P(C|E).

So if the BA framework is to give the correct answer, we need to have P(x|EC) ∝ P(x|E)P(x|C), so that the normalisation in the BA framework fixes everything correctly. Since P(C|E) is also just a normalising factor, this proportionality occurs if and only if P(C|xE) ∝ P(x|C), which does not hold in general. In the precise setting that was originally proposed for the BA framework, there are two special features. Firstly, the estimate is a noisy measurement of x, and so P(C|x) = P(C|xE) because all dependence on the world factors through x. Secondly P(C|x) ∝ P(x|C), and so the bayesian and BA results coincide.

However, when we investigate an indirect intervention we are typically looking at estimates derived non-trivially from the world; as a result, P(C|xE) ≠ P(C|x), and the BA framework breaks down. Put another way, when we look for estimates and find one, we have learned something about the world. If we don't account for this properly, we will make incorrect conclusions.

In particular, it is reasonable to expect that the existence of estimates implying unusual values for an intervention should positively correlate with background states of the world which permit unusual values for the intervention. The BA framework does not account for this, and so heuristically it will overly penalise estimates of interventions which yield results far from the prior distribution. Of course, we can reasonably ask whether it is feasible to compute P(x|EC) explicitly; in general fully bayesian work is hard.

Jaynes (Probability Theory: The Logic of Science, Chapter 18) deals with a simpler example of the same basic problem, where we are asked to ascribe credence to a proposition like

A := “when I flip this coin it will come up heads”.

Instead of merely having a belief about the distribution over outcomes (analogous to P(x|E) in the BA case), it turns out to be necessary to keep track of a distribution over propositions of form:

A_p := “the subjective probability of Heads is p, regardless of any other evidence”;

or more formally we define P(A|A_pE) = p. Hence the events A_p are disjoint, and exactly one is true. Hence we have an object which behaves like a probability distribution over A_p; we can abuse terminology and use probability directly. Jaynes then shows that:

P(A) = ∫p P(A_p) dp

And so we can recover P(A) from the P(A_p). The full A_p distribution is needed to formalise confidence in one’s estimate. For example, if one is sure from background data E that the coin is completely biased, then one trial flip will tell you which way the coin is biased, and so P(A|E,F) will be almost 0 or 1, whilst P(A|E) = ½. On the other hand, if you have background information E’ that of 10000 trial flips 5000 were heads, then one additional trial flip F leaves P(A|E’F) ~ P(A|E) = ½. Jaynes shows that the A_p distribution screens off E, and can be updated in light of new data F; the posterior P(A|EF) is then the mean of the new A_p distribution. In this framework, and starting from a uniform prior over A_p, Laplace’s law of succession is derived.

To generalise this framework to estimating a real value x rather than a binary outcome A, we can shift from a distribution A_p over probabilities of A to a distribution P(X_d) over distributions for x, with X_d := "x ~ d regardless of other evidence"¹. In this setting, there will still be a “point estimate” distribution X, the mean of X_d, which summarises your current beliefs about x. Other information about X_d is needed to allow you to update coherently in response to arbitrary new information. In such a case, new information may cause one to substantially change the distribution X_d, and thus one’s beliefs about the world, if this new information causes a great deal of surprise conditional on X.

Examples and Priors in the BA framework

The mathematics can also reveal when an intuition pump is bringing extra information in a non-obvious way. For example, some of the examples given for how the BA framework should run had the apparently unintuitive feature that successively larger claims of impact eventually lead to decreasing posterior means to the estimates. This turns out to be because the standard deviation of the estimates was presumed to be roughly equal to their mean.

De facto this means that the new evidence was prohibited a prior from suggesting that an intervention was better than the prior mean with high probability. In general, this need not hold, if we are able to find data which is reasonably constrained and not present in the background model. If we intend to also account for the possibilities of errors in cognition, then this kind of treatment of new evidence seems more reasonable, but then we should see similar broadening in our background prior.

Similarly, as the stated BA priors are normal or log-normal, they assert that the event E := “the range of intervention impact ratios is large” has very low probability. Some decay is necessary to prevent arbitrarily large impacts dominating, which would make expected value computations fail to converge. Practically, this implies that a stated prior for impacts drops off faster than 1/impact³ above some impact², but this does not in and of itself mandate a specific form of prior, not specify the point above which the prior should drop rapidly, nor the absolute rate of the drop off. In particular, the log-normal or normal prior drop off much faster, and so are implicitly very confident that the range of impacts is bounded by what we've already seen.

What is the range of impacts for interventions?

It is not trivial to find out what kinds of ratios we should expect to see; for these purposes it is unfortunate that Givewell does not publicly emit $/DALY or $/life estimates of impact for the majority of the charities it assesses. It would be very useful to see what kinds of impacts are being sampled at the low end. Other studies (eg. DCP2) have assessed some hundreds of high and low impact interventions in public health, and assert 10000:1 ratios in impact, with their best $/DALY numbers consistent with Givewell’s assessment that AMF is likely to be one of the better public health interventions available.

Of course, we also strongly suspect that there exist interventions with better impacts than AMF, if we are willing to look outside public health. Givewell raison d’etre is that one can gain leverage in moving funds from ineffective causes to effective ones, and so a dollar spent on Givewell should move much more than a dollar to effective interventions. In principle this demonstrates that the range of possible intervention impacts may be much larger than the range available in specific fields, such as developing world health interventions.

By the lights of the BA prior, we are uncharitable about an estimate of impact if we assert it is large, in that this makes the estimate incredulous and thus heavily discounted. In this sense, existential risk reduction has been sketchily and optimistically estimated at around $0.125/life, which we can take as an uncharitable estimate for the BA framework. Assuming that this was a correct estimate, it being true would only require the existence of an intervention which is to AMF as AMF is to the least effective health interventions. It does not seem easy to confidently assert that the tail thickness and variance of the distribution of intervention impacts is such that the apparently observed ratios in public health interventions and Givewell are common enough that they can be searched for whilst ruling out a priori the credibility of estimates at the <$1/life level.

Now, it might be possible that these very high impact interventions are not easy to scale up, or are rare enough that it is not worth searching for them. On the other hand, we can free-ride on other people recommending interventions, if we are willing to accept internal or inside view assessments as substantively credible.

Confidence and Probability

It seems clear that the probability of a proposition and one’s confidence in the quality of your assessment are distinct, although it is easy to confuse language by referring to confidence in a proposition, rather than in a probability or estimate. Fully rigorously, this is encompassed in the distribution over X_d, but in practise we may wish to track only a single posterior distribution³.

Other commenters have suggested a similar distinction between confidence and probability; observing that having observed the computations exist the correct response is to say “I notice that I am confused”. More formally, in practise we have neither P(x|C) nor P(x|E). We have to also condition on some event like:

N := "My modelling and computations are correct".

Ideally one would have extensive tests of all of the pieces of a methodology, so that one could say something about which classes of interventions are well modelled, but practically this may excessively complicate the issue. A priori, it seems unreasonable to attach >> 1-1/1000 probability to propositions like N for a new method or model which has merely been output by human cognition. Assessing high confidence would be expected to wait on assessing the reliability and calibration of the methodology, or showing that the model is a stable output of cognition.

In the event of a computation and a point prior belief about interventions disagreeing, a Bayesian update will reduce confidence in N, and also come to believe that the processes leading to the estimate C are less reliable. This is separate to the process which causes you to extract beliefs about this particular intervention. Whether the background model is substantively changed or the estimation procedure is discounted is a matter for your relative confidence in these processes, and the sensitivity of the outputs of the processes.

Conclusions

Disagreements over how to estimate the impact of an intervention on the world have existed for some time, and it seems that the grounds for these disagreements are not being well addressed. In general, it would be a good thing for our grounds for confidence in arguments and background priors to be made very explicit and open. In principle we can then reduce these disagreements to matters of fact and differences in prior beliefs.

In the particular case of Givewell, it is clear that they have assessed a great many interventions systematically, and seem to possess a great deal of confidence in their modelled backgrounds. I do not know if there has been a formal process of checking the calibration of these estimates; if there has been, and so Givewell can assess in high confidence (say » 10 bits) in propositions of form “our model is emitting a suitable background correct for this class of interventions”, then the methods are highly likely to be highly valuable to the wider EA community for other purposes, and ideally would be distributed.

Notes

I wrote this post whilst a visiting fellow at MIRI; Lukeprog asked that I take a further look at LW's debates on cost effectiveness stability in effective altruism, and try to clarify the situation if possible.

I am grateful to Carl Shulman, Luke Muehlhauser and Adam Casey for their substantive feedback and comments on early drafts of this post.

1 To follow the modified mathematics of Jaynes' derivation closely, we amend 18-1 to read P(X = x|X_dE) = d(x) for any distribution d, and then follow Jaynes’ derivation formally. It is reasonable to be worried that the space of distributions is not measurable; this can be fixed by restricting to a sigma-algebra of functions which are piecewise constant (or alternatively running Jaynes' original approach on the set of binary propositions A_yz := "y ≤ x ≤ z" for all y and z)

2 We could also assert strong cancellation properties, but it is unclear whether these effects can be substantial in practise. Technically, we also could get convergence with drop offs like 1/(n² log² n) or 1/(n² log n log² log n), but the distinction is slight for the purposes of discussion; they are much slower than a normal.

3 If we work with the set of A_xy propositions instead, then Jaynes implies we have to hold a set of distributions (A_xy)_p, which is rather more tractable than X_d, although harder to visualise concretely.

5 comments

Tiling Agents for Self-Modifying AI (OPFAI #2)

Thu, 06 Jun 2013 20:24:25 +0000

Submitted by Eliezer_Yudkowsky • 51 votes • 179 comments

An early draft of publication #2 in the Open Problems in Friendly AI series is now available: Tiling Agents for Self-Modifying AI, and the Lobian Obstacle. ~20,000 words, aimed at mathematicians or the highly mathematically literate. The research reported on was conducted by Yudkowsky and Herreshoff, substantially refined at the November 2012 MIRI Workshop with Mihaly Barasz and Paul Christiano, and refined further at the April 2013 MIRI Workshop.

Abstract:

We model self-modication in AI by introducing 'tiling' agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of the offspring's goals). Constructing a formalism in the most straightforward way produces a Godelian difficulty, the Lobian obstacle. By technical methods we demonstrate the possibility of avoiding this obstacle, but the underlying puzzles of rational coherence are thus only partially addressed. We extend the formalism to partially unknown deterministic environments, and show a very crude extension to probabilistic environments and expected utility; but the problem of finding a fundamental decision criterion for self-modifying probabilistic agents remains open.

Commenting here is the preferred venue for discussion of the paper. This is an early draft and has not been reviewed, so it may contain mathematical errors, and reporting of these will be much appreciated.

The overall agenda of the paper is introduce the conceptual notion of a self-reproducing decision pattern which includes reproduction of the goal or utility function, by exposing a particular possible problem with a tiling logical decision pattern and coming up with some partial technical solutions. This then makes it conceptually much clearer to point out the even deeper problems with "We can't yet describe a probabilistic way to do this because of non-monotonicity" and "We don't have a good bounded way to do this because maximization is impossible, satisficing is too weak and Schmidhuber's swapping criterion is underspecified." The paper uses first-order logic (FOL) because FOL has a lot of useful standard machinery for reflection which we can then invoke; in real life, FOL is of course a poor representational fit to most real-world environments outside a human-constructed computer chip with thermodynamically expensive crisp variable states.

As further background, the idea that something-like-proof might be relevant to Friendly AI is not about achieving some chimera of absolute safety-feeling, but rather about the idea that the total probability of catastrophic failure should not have a significant conditionally independent component on each self-modification, and that self-modification will (at least in initial stages) take place within the highly deterministic environment of a computer chip. This means that statistical testing methods (e.g. an evolutionary algorithm's evaluation of average fitness on a set of test problems) are not suitable for self-modifications which can potentially induce catastrophic failure (e.g. of parts of code that can affect the representation or interpretation of the goals). Mathematical proofs have the property that they are as strong as their axioms and have no significant conditionally independent per-step failure probability if their axioms are semantically true, which suggests that something like mathematical reasoning may be appropriate for certain particular types of self-modification during some developmental stages.

Thus the content of the paper is very far off from how a realistic AI would work, but conversely, if you can't even answer the kinds of simple problems posed within the paper (both those we partially solve and those we only pose) then you must be very far off from being able to build a stable self-modifying AI. Being able to say how to build a theoretical device that would play perfect chess given infinite computing power, is very far off from the ability to build Deep Blue. However, if you can't even say how to play perfect chess given infinite computing power, you are confused about the rules of the chess or the structure of chess-playing computation in a way that would make it entirely hopeless for you to figure out how to build a bounded chess-player. Thus "In real life we're always bounded" is no excuse for not being able to solve the much simpler unbounded form of the problem, and being able to describe the infinite chess-player would be substantial and useful conceptual progress compared to not being able to do that. We can't be absolutely certain that an analogous situation holds between solving the challenges posed in the paper, and realistic self-modifying AIs with stable goal systems, but every line of investigation has to start somewhere.

Parts of the paper will be easier to understand if you've read Highly Advanced Epistemology 101 For Beginners including the parts on correspondence theories of truth (relevant to section 6) and model-theoretic semantics of logic (relevant to 3, 4, and 6), and there are footnotes intended to make the paper somewhat more accessible than usual, but the paper is still essentially aimed at mathematically sophisticated readers.

179 comments

Newest Submissions - Less Wrong

Some reservations about Singer's child-in-the-pond argument

Giving Now Currently Seems to Beat Giving Later

Introduction

But First, Why Give Later?

Ok, So What's The Investment Rate, Then?

Ok, So What's The Growth Rate, Then?

Other Complications

Weekly LW Meetups

Effective Altruism Through Advertising Vegetarianism?

Introduction

Other Estimations

Pamphlets Per Dollar

Conversions Per Pamphlet

Facebook Study

Pamphlet Study

Vegetarian Years Per Conversion

Animals Saved Per Vegetarian Year

Days Lived Per Animal

Accounting For Biases

Additional People Are Being Reached

Accounting for Product Elasticity

Putting It All Together

Market Saturation and Diminishing Marginal Returns?

The Conjunction Fallacy?

Conversion and The 100 Yard Line

Three Places I Might Donate Before Donating to Vegan Outreach

Conclusion

Footnotes

[3]: I calculated the number of vegetarians by eyeballing about how many people said they no longer eat fish, which I'd guess only a vegetarian would be willing to give up.

[4]: 32 vegetarians / 104 people = 30.7%. That population is 8.5% (7% for likes + 1.5% for the starter kit) of the overall population, leading to 2.61% (30.7% * 8.5%).

[7]: Feel free to check the formula for accuracy and also check to make sure the calculator implements the formula correctly. I worry that the added accuracy from the complex calculator is outweighed by the risk that the formula is wrong.

A personal history of involvement with effective altruism

Interest in altruism rooted in literature

An analytical bent, and utilitarianism

Epistemic paralysis

Enter GiveWell

The significance of Less Wrong

Closing summary

Anticipating critical transitions

Robust Cooperation in the Prisoner's Dilemma

Background: Writing programs to play the PD with source code swap

Previously known: CliqueBot and FairBot

Generalization: Modal Agents

PrudentBot: The best of both worlds?

Results: Obstacles to Optimality

Why is this awesome, and what's next?

Further Reading on LessWrong

Prisoner's Dilemma (with visible source code) Tournament

Model Stability in Intervention Assessment

Bayesian Adjustment, and the Ap distribution

Examples and Priors in the BA framework

What is the range of impacts for interventions?

Confidence and Probability

Conclusions

Notes

Tiling Agents for Self-Modifying AI (OPFAI #2)

Bayesian Adjustment, and the A_p distribution