Comment Permalink

Jonah,

Thanks for expressing an interest in donating to SIAI.

(a) SIAI has secured a 2 star donation from GiveWell for donors who are interested in existential risk.

I assure you that we are very interested in getting the GiveWell stamp of approval. Michael Vassar and Anna Salamon have corresponded with Holden Karnofsky on the matter and we're trying figure out the best way to proceed.

If it were just a matter of SIAI becoming more transparent and producing a larger number of clear outputs I would say that it is only a matter of time. As it stands, GiveWell does not know how to objectively evaluate activities focused on existential risk reduction. For that matter, neither do we, at least not directly. We don't know of any way to tell what percentage of worlds that branch off from this one go on to flourish and how many go on to die. If GiveWell decides not to endorse charities focused on existential risk reduction as a general policy, there is little we can do about it. Would you consider an alternative set of criteria if this turns out to be the case?

We think that UFAI is the largest known existential risk and that the most complete solution - FAI - addresses all other known risks (as well as the goals of every other charitable cause) as a special case. I don't mean to imply that AI is the only risk worth addressing at the moment, but it certainly seems to us to be the best value on the margin. We are working to make the construction of UFAI less likely through outreach (conferences like the Summit, academic publications, blog posts like The Sequences, popular books and personal communication) and make the construction of FAI more likely through direct work on FAI theory and the identification and recruitment of more people capable of working on FAI. We've met and worked with several promising candidates in the past few months. We'll be informing interested folk about our specific accomplishments in our new monthly newsletter, the June/July issue of which was sent out a few weeks ago. You can sign up here.

(b) You publically apologize for and qualify your statements quoted by XiXiDu here. I believe that these statements are very bad for public relations. Even if true, they are only true at the margin and so at very least need to be qualified in that way.

It would have been a good idea for you to watch the videos yourself before assuming that XiXiDu's summaries (not actual quotes, despite the quotation marks that surrounded them) were accurate. Eliezer makes it very clear, over and over, that he is speaking about the value of contributions at the margin. As others have already pointed out, it should not be surprising that we think the best way to "help save the human race" is to contribute to FAI being built before UFAI. If we thought there was another higher-value project then we would be working on that. Really, we would. Everyone at SIAI is an aspiring rationalist first and singularitarian second.

multifoliaterose14y120

If GiveWell decides not to endorse charities focused on existential risk reduction as a general policy, there is little we can do about it. Would you consider an alternative set of criteria if this turns out to be the case?

Yes, I would consider an alternative set of criteria if this turns out to be the case.

I have long felt that GiveWell places too much emphasis on demonstrated impact and believe that in doing so GiveWell may be missing some of the highest expected value opportunities for donors.

It would have been a good idea for you to watch the vi

... (read more)

See in context

20 Transparency and Accountability

by multifoliaterose

21st Aug 2010

6 min read

145

20

[Added 02/24/14: After writing this post, I discovered that I had miscommunicated owing to not spelling out my thinking in sufficient detail, and also realized that it carried unnecessary negative connotations (despite conscious effort on my part to avoid them). See Reflections on a Personal Public Relations Failure: A Lesson in Communication. SIAI (now MIRI) has evolved substantially since 2010 when I wrote this post, and the criticisms made in the post don't apply to MIRI as presently constituted.]

Follow-up to: Existential Risk and Public Relations, Other Existential Risks, The Importance of Self-Doubt

Over the last few days I've made a string of posts levying strong criticisms against SIAI. This activity is not one that comes naturally to me. In The Trouble With Physics Lee Smolin writes

...it took me a long time to decide to write this book. I personally dislike conflict and confrontation [...] I kept hoping someone in the center of string-theory research would write an objective and detailed critique of exactly what has and has not been acheived by the theory. That hasn't happened.

My feelings about and criticisms of SIAI are very much analogous to Smolin's feelings about and criticisms of string theory. Criticism hurts feelings and I feel squeamish about hurting feelings. I've found the process of presenting my criticisms of SIAI emotionally taxing and exhausting. I fear that if I persist for too long I'll move into the region of negative returns. For this reason I've decided to cut my planned sequence of posts short and explain what my goal has been in posting in the way that I have.

Edit: Removed irrelevant references to VillageReach and StopTB, modifying post accordingly.

As Robin Hanson never ceases to emphasize, there's a disconnect between what humans say that what they're trying to do and what their revealed goals are. Yvain has written about this topic recently under his posting Conflicts Between Mental Subagents: Expanding Wei Dai's Master-Slave Model. This problem becomes especially acute in the domain of philanthropy. Three quotes on this point:

(1) In Public Choice and the Altruist's Burden Roko says:

The reason that we live in good times is that markets give people a selfish incentive to seek to perform actions that maximize total utility across all humans in the relevant economy: namely, they get paid for their efforts. Without this incentive, people would gravitate to choosing actions that maximized their own individual utility, finding local optima that are not globally optimal. Capitalism makes us all into efficient little utilitarians, which we all benefit enormously from.

The problem with charity, and especially efficient charity, is that the incentives for people to contribute to it are all messed up, because we don't have something analogous to the financial system for charities to channel incentives for efficient production of utility back to the producer.

(2) In My Donation for 2009 (guest post from Dario Amodei) Dario says:

I take Murphy’s Law very seriously, and think it’s best to view complex undertakings as going wrong by default, while requiring extremely careful management to go right. This problem is especially severe in charity, where recipients have no direct way of telling donors whether an intervention is working.

(3) In private correspondence about career choice, Holden Karnofsky said:

For me, though, the biggest reason to avoid a job with low accountability is that you shouldn't trust yourself too much. I think people respond to incentives and feedback systems, and that includes myself. At GiveWell I've taken some steps to increase the pressure on me and the costs of I behave poorly or fail to add value. In some jobs (particularly in nonprofit/govt) I feel that there is no system in place to help you figure out when you're adding value and incent you to do so. That matters a lot no matter how altruistic you think you are.

I believe that the points that Robin, Yvain, Roko, Dario and Holden have made provide a compelling case for the idea that charities should strive toward transparency and accountability. As Richard Feynman has said:

The first principle is that you must not fool yourself – and you are the easiest person to fool.

Because it's harder to fool others than it is to fool oneself, I think that the case for making charities transparent and accountable is very strong.

SIAI does not presently exhibit high levels of transparency and accountability. I agree with what I interpret to be Dario's point above: that in evaluating charities which are not transparent and accountable, we should assume the worst. For this reason together with the concerns which I express about Existential Risk and Public Relations, I believe that saving money in a donor-advised-fund with a view toward donating to a transparent and accountable future existential risk organization has higher expected value than donating to SIAI now does.

Because I take astronomical waste seriously and believe in shutting up and multiplying, I believe that reducing existential risk is ultimately more important than developing world aid. I would very much like it if there were a highly credible existential risk charity. At present, I do not feel that SIAI is a credible existential risk charity. One LW poster sent me a private message saying:

I've suspected for a long time that the movement around EY might be a sophisticated scam to live off donations of nonconformists

I do not believe that Eliezer is consciously attempting to engage in a scam to live off of the donations but I believe that (like all humans) he is subject to subconscious influences which may lead him to act as though he were consciously running a scam to live off of the donations of nonconformists. In light of Hanson's points, it would not be surprising if this were the case. The very fact that I received such a message is a sign that SIAI has public relations problems.

I encourage LW posters who find this post compelling to visit and read the materials available at GiveWell which is, as far as I know, the only charity evaluator which places high emphasis on impact, transparency and accountability. I encourage LW posters who are interested in existential risk to contact GiveWell expressing interest in GiveWell evaluating existential risk charities. I would note that it may be useful for LW posters who are interested in finding transparent and accountable organizations to donate to GiveWell's recommended charities to signal seriousness to the GiveWell staff.

I encourage SIAI to strive toward greater transparency and accountability. For starters, I would encourage SIAI to follow the example set by GiveWell and put a page on its website called "Mistakes" publically acknowledging its past errors. I'll also note that GiveWell incentivizes charities to disclose failures by granting them a 1-star rating. As Elie Hassenfeld explains

As usual, we’re not looking for marketing materials, and we won’t accept “weaknesses that are really strengths” (or reports that blame failure entirely on insufficient funding/support from others). But if you share open, honest, unadulterated evidence of failure, you’ll join a select group of organizations that have a GiveWell star.

I believe that the fate of humanity depends on the existence of transparent and accountable organizations. This is both because I believe that transparent and accountable organizations are more effective and because I believe that people are more willing to give to them. As Holden says:

I must say that, in fact, much of the nonprofit sector fits incredibly better into Prof. Hanson’s view of charity as “wasteful signaling” than into the traditional view of charity as helping.

[...]

Perhaps ironically, if you want a good response to Prof. Hanson’s view, I can’t think of a better place to turn than GiveWell’s top-rated charities. We have done the legwork to identify charities that can convincingly demonstrate positive impact. No matter what one thinks of the sector as a whole, they can’t argue that there are no good charitable options - charities that really will use your money to help people - except by engaging with the specifics of these charities’ strong evidence.

Valid observations that the sector is broken - or not designed around helping people - are no longer an excuse not to give.

Because our Bayesian prior is so skeptical, we end up with charities that you can be confident in, almost no matter where you’re coming from.

I believe that at present the most effective way to reduce existential risk is to work toward the existence of a transparent and accountable existential risk organization.

Added 08/23:

See my responses to Yvain's comment for my answer to Yvain's question "...why the feeling that they're not transparent and accountable?"
See my response to Jordan's comment for an explanation of why I did not discuss FHI in the present posting.

Personal Blog

20

Mentioned in

50Reflections on a Personal Public Relations Failure: A Lesson in Communication

Transparency and Accountability

New Comment

145 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:30 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Airedale14y390

Your posts on SIAI have had a veneer of evenhandedness and fairness, and that continues here. But given what you don’t say in your posts, I cannot avoid the impression that you started out with the belief that SIAI was not a credible charity and rather than investigating the evidence both for and against that belief, you have marshaled the strongest arguments against donating to SIAI and ignored any evidence in favor of donating to SIAI. I almost hesitate to link to EY lest you dismiss me as one of his acolytes, but see, for example, A Rational Argument.

In your top-level posts you have eschewed references to any of the publicly visible work that SIAI does such as the Summit and the presentation and publication of academic papers. Some of this work is described at this link to SIAI’s description of its 2009 achievements. The 2010 Summit is described here. As for Eliezer’s current project, at the 2009 achievements link, SIAI has publicized the fact that he is working on a book on rationality:

Yudkowsky is now converting his blog sequences into the planned rationality book, which he hopes will significantly assist in attracting and inspiring talented individuals to effectively

... (read more)

3XiXiDu14y

"If you’re interested in being on the right side of disputes, you will refute your opponents’ arguments. But if you’re interested in producing truth, you will fix your opponents’ arguments for them. To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse." -- Black Belt Bayesian If multifoliaterose took the position of a advocatus diaboli, what would be wrong with that?

9Airedale14y

Although I always love a good quote from Black Belt Bayesian (a/k/a steven0461 a/k/a my husband), I think he’s on board with my interpretation of multifoliaterose’s posts. (At least, he’d better be!) Going on to the substance, it doesn’t seem that multifoliaterose is just playing devil’s advocate here rather than arguing his actual beliefs – indeed everything he’s written suggests that he’s doing the latter. Beyond that, there may be a place for devil’s advocacy (so long as it doesn’t cross the line into mere trolling, which multifoliaterose’s posts certainly do not) at LW. But I think that most aspiring rationalists (myself included) should still try to evaluate evidence for and against some position, and only tread into devil’s advocacy with extreme caution, since it is a form of argument where it is all too easy to lose sight of the ultimate goal of weighing the available evidence accurately.

4XiXiDu14y

Wow, I managed to walk into the lion's den there! Yeah, I wasn't actually thinking that to be the case either. But since nobody else seems to be following your husbands advice...at least someone tries to argue against the SIAI. Good criticism can be a good thing. I see, I'll take your word for it. I haven't thought about it too much. So far I thought your husbands quote is universally applicable.

2wedrifid14y

Multi has already refuted the opponent's arguments, well at least Eliezer more or less refuted them for him. Now it is time to do just what Black Belt Bayesian suggested and try to fix the SIAI's arguments for them. Because advocacy - including devil's advocacy - is mostly bullshit. Remind SIAI of what they are clearly doing right and also just what a good presentation of their strengths would look like - who knows, maybe it'll spur them on and achieve in some measure just the kind of changes you desire!

5XiXiDu14y

Interesting! Levels of epistemic accuracy: * (1) Truth * (2) Concealment * (3) Falsehood * (4) Bullshit (Not even wrong) So while telling the truth is maximally accurate relative to your epistemic state, concealment is deception by misguidance which is worse than the purest form of deception that is lying (falsehood). Bullshit however is not even wrong. I don't see how devil's advocacy fits into this as I perceive it to be a temporary adjustment of someones mental angel to look back at one's own position from a different point of view.

0multifoliaterose14y

See my response to Jordan's comment.

0multifoliaterose14y

Hi Airedale, Thanks for your thoughtful comments. I'm missing a good keyboard right now so can't respond in detail, but I'll make a few remarks. I'm well aware that SIAI has done some good things. The reason why I've focusing on the apparent shortcomings of SIAI is to encourage SIAI to improve its practices. I do believe that at the margin the issue worthy of greatest consideration is transparency and accountability and I believe that this justifies giving to VillageReach over SIAI. But I'm definitely open to donating to and advocating that others donate to SIAI and FHI in the future provided that such organizations clear certain standards for transparency and accountability and provide a clear and compelling case for room for more funding. Again, I would encourage you (and others) who are interested in existential risk to write to the GiveWell staff requesting that GiveWell evaluate existential risk organizations including SIAI and FHI. I would like to see GiveWell do such work soon.

[-]WrongBot14y100

I do believe that at the margin the issue worthy of greatest consideration is transparency and accountability and I believe that this justifies giving to VillageReach over SIAI.

What about everything else that isn't the margin? What is your expected value of SIAI's public accomplishments, to date, in human lives saved? What is that figure for VillageReach? Use pessimistic figures for SIAI and optimistic ones for VillageReach if you must, but come up with numbers and then multiply them. Your arguments are not consistent with expected utility maximization.

You would be much better off if you were directly offering SIAI financial incentives to improve the expected value of its work. Donating to VillageReach is not the optimal use of money for maximizing what you value.

3multifoliaterose14y

You may well be right about this, I'll have to think some more about this :-). Thanks for raising this issue.

9Jordan14y

You've provided reasons for why you are skeptical of the ability of SIAI to reduce existential risk. It's clear you've dedicated a good amount of effort to your investigation. Why are you content to leave the burden of investigating FHI's abilities to GiveWell, rather than investigate yourself, as you have with SIAI?

1multifoliaterose14y

The reason that I have not investigated FHI is simply because I have not gotten around to doing so. I do plan to change this soon. I investigated SIAI first because I came into contact with SIAI before I came into contact with FHI. My initial reaction to FHI is that it looks highly credible to me, but that I doubt that it has room for more funding. However, I look forward to looking more closely into this matter in the hopes of finding a good opportunity for donors to lower existential risk.

7Airedale14y

You should definitely do research to confirm this on your own, but the last I heard (somewhat informally through the grapevine) was that FHI does indeed have room for more funding, for example, in the form of funding for an additional researcher or post-doc to join their team. You can than evaluate whether an additional academic trying to research and publish in these areas would be useful, but given how small the field currently is, my impression would be that an additional such academic would probably be helpful.

1multifoliaterose14y

Thanks for the info.

3wedrifid14y

I would very much like to see what positive observations you have made during your research into the SIAI. I know that you believe there is plenty of potential - there would be no reason to campaign for improvements if you didn't see the chance that it would make a difference. That'd be a pointless or counter-productive for your interests given that it certainly doesn't win you any high status friends! How about you write a post on a different issue regarding SIAI or FAI in general using the same standard of eloquence that you have displayed?

2multifoliaterose14y

wedrifid - Thanks for your kind remarks. As I said in my top level post, I'll be taking a long break from LW. As a brief answer to your question: (a) I think that Eliezer has inspired people (including myself) to think more about existential risk and that this will lower existential risk. I thank Eliezer for this. (b) I think that Less Wrong has provided a useful venue for smart people (of a certain kind) to network and find friends and that this too will lower existential risk. (c) Most of what I know about the good things that SIAI has done on an institutional level are from Carl Shulman. You might like to ask him for more information.

[-]Scott Alexander14y250

The bulk of this is about a vague impression that SIAI isn't transparent and accountable. You gave one concrete example of something they could improve: having a list of their mistakes on their website. This isn't a bad idea, but AFAIK GiveWell is about the only charity that currently does this, so it doesn't seem like a specific failure on SIAI's part not to include this. So why the feeling that they're not transparent and accountable?

SIAI's always done a good job of letting people know exactly how it's raising awareness - you can watch the Summit videos yourself it you want. They could probably do a bit more to publish appropriate financial records, but I don't think that's your real objection. Besides that, what? Anti-TB charities can measure how much less TB there is per dollar invested; SIAI can't measure what percentage safer the world is, since the world-saving is still in basic research phase. You can't measure the value of the Manhattan Project in "cities destroyed per year" while it's still going on.

By the Outside View, charities that can easily measure their progress with statistics like "cases of TB prevented" are better than those that can't. By the... (read more)

5multifoliaterose14y

In addition to the points that I made in my other response to your comment, I would add that the SIAI staff have not created an environment which welcomes criticism from outsiders. The points in my other response were well considered and yet as I write, the response has been down voted three times so that it is now hidden from view. I see Eliezer's initial response to XiXiDu's post Should I believe what the SIAI claims? as evidence that the SIAI staff have gotten in the habit of dismissing criticisms out of hand whenever they question the credibility of the critic. This habit is justified up to a point, but my interactions with the SIAI staff have convinced me that (as a group) they go too way far, creating a selection effect which subjects themselves to confirmation bias and group think. In SIAI's defense, I'll make the following three points: •Michael Vassar corresponded with me extensively in response to the criticisms of SIAI which I raised in the comments to my post (One reason) why capitalism is much maligned and Roko's post Public Choice and the Altruist's Burden. •As prase remarks, the fact that my top level posts have not been censored are an indication that "LW is still far from Objectivism." •SIAI staff member Jasen sent me a private message thanking me for making my two posts Existential Risk and Public Relations and Other Existential Risks and explaining that SIAI plans to address these points in the future. I see these things as weak indications that SIAI may take my criticisms seriously. Nevertheless, I perceive SIAI's openness to criticism up until this point to be far lower than GiveWell's openness to criticism up until this point. For examples of GiveWell's openness to criticism, see Holden Karnofsky's back and forth with Laura Deaton here and the threads on the GiveWell research mailing list on Cost-Effectiveness Estimates, Research Priorities and Plans and Environmental Concerns and International Aid as well as Holden's posting Population

5multifoliaterose14y

Yvain, Thanks for your feedback. •As I discuss in Other Existential Risks I feel that SIAI has not (yet) provided a compelling argument for the idea that focus on AI is the most cost-effective way of reducing existential risk. Obviously I don't expect an airtight argument (as it would be impossible to offer one), but I do feel SIAI needs to say a lot more on this point. I'm encouraged that SIAI staff have informed me that more information on this point will be forthcoming in future blog posts. •I agree with SarahC's remarks here and here on the subject of there being a "problem with connecting to the world of professional science." •I agree with you that SIAI and its strategy aren't as crazy as they sound at first blush. I also agree that the Less Wrong sequences suffice to establish that SIAI has some people of very high intellectual caliber and that this distinguishes SIAI from most charities. At present, despite these facts I'm very skeptical of the idea that SIAI's approach to reducing existential risk is the optimal one for the reasons given in the two bullet points above. •Regarding: Here the question is just whether there's sufficient evidence that donating to SIAI has sufficiently high (carefully calibrated) expected value to warrant ignoring incentive effects associated with high focus on transparency and accountability. I personally am very skeptical that this question has an affirmative answer. I may be wrong.

[-]CarlShulman14y230

I believe that at present GiveWell's top ranked charities VillageReach and StopTB are better choices than SIAI, even for donors like utilitymonster who take astronomical waste seriously and believe in the ideas expressed in the cluster of blog posts linked under Shut Up and multiply.

The invocation of VillageReach in addressing those aggregative utilitarians concerned about astronomical waste here seems baffling to me. Consider these three possibilities:

1) SIAI at the margin has a negative expected impact on our chances of avoiding existential risks, so shouldn't be donated to. VillageReach is irrelevant and adds nothing to the argument, you could have said "aggregative utilitarians would do better to burn their cash." Why even distract would-be efficient philanthropists with this rather than some actual existential-risk-focused endeavour, e.g. FHI, or a donor-advised fund for existential risk, or funding a GiveWell existential risk program, or conditioning donations based on some transparency milestones?

2) SIAI at the margin has significant positive expected impact on our chances of avoiding existential risks. VillageReach may very slightly and indirectly reduce existe... (read more)

[-]multifoliaterose14y130

Your points are fair, I have edited the top level post accordingly to eliminate reference to VillageReach and StopTB.

[-]Eliezer Yudkowsky14y190

SIAI does not presently exhibit high levels of transparency and accountability... For this reason together with the concerns which I express about Existential Risk and Public Relations, I believe that at present GiveWell's top ranked charities VillageReach and StopTB are better choices than SIAI

I have difficulty taking this seriously. Someone else can respond to it.

agree with what I interpret to be Dario's point above: that in evaluating charities which are not transparent and accountable, we should assume the worst.

Assuming that much of the worst isn't rational. It would be a convenient soldier for your argument, but it's not the odds to bet at. Also, you don't make clear what constitutes a sufficient level of transparency and accountability, though of course you will now carefully look over all of SIAI's activities directed at transparency and accountability, and decide that the needed level is somewhere above that.

You say you assume the worst, and that other people should act accordingly. Would you care to state "the worst", your betting odds on it, how much you're willing to bet, and what neutral third party you would accept as providing the verdict if they... (read more)

[-]Jordan14y120

Normally I refrain from commenting about the tone of a comment or post, but the discussion here revolves partly around the public appearance of SIAI, so I'll say:

this comment has done more to persuade me to stop being a monthly donor to SIAI than anything else I've read or seen.

This isn't a comment about the content of your response, which I think has valid points (and which multifoliaterose has at least partly responded to).

[-]wedrifid14y280

this comment has done more to persuade me to stop being a monthly donor to SIAI than anything else I've read or seen.

It is certainly Eliezer's responses and not multi's challenges which are the powerful influence here. Multi has effectively given Eliezer a platform from which to advertise the merits of SIAI as well as demonstrate that contrary to suspicions Eliezer is, in fact, able to handle situations in accordance to his own high standards of rationality despite the influences of his ego. This is not what I've seen recently. He has focussed on retaliation against multi at whatever weak points he can find and largely neglected to do what will win. Winning in this case would be demonstrating exactly why people ought to trust him to be able to achieve what he hopes to achieve (by which I mean 'influence' not 'guarantee' FAI protection of humanity.)

I want to see more of this:

With Michael Vassar in charge, SIAI has become more transparent, and will keep on doing things meant to make it more transparent

and less of this:

I have to say that my overall impression here is of someone who manages to talk mostly LW language most of the time, but when his argument requires a step that

... (read more)

[-]Eliezer Yudkowsky14y690

This sort of conversation just makes me feel tired. I've had debates before about my personal psychology and feel like I've talked myself out about all of them. They never produced anything positive, and I feel that they were a bad sign for the whole mailing list they appeared on - I would be horrified to see LW go the way of SL4. The war is lost as soon as it starts - there is no winning move. I feel like I'm being held to an absurdly high standard, being judged as though I were trying to be the sort of person that people accuse me of thinking I am, that I'm somehow supposed to produce exactly the right mix of charming modesty while still arguing my way into enough funding for SIAI... it just makes me feel tired, and like I'm being held to a ridiculously high standard, and that it's impossible to satisfy people because the standard will keep going up, and like I'm being asking to solve PR problems that I never signed up for. I'll solve your math problems if I can, I'll build Friendly AI for you if I can, if you think SIAI needs some kind of amazing PR person, give us enough money to hire one, or better yet, why don't you try being perfect and see whether it's as easy as it so... (read more)

[-]wedrifid14y250

I really appreciate this response. In fact, to mirror Jordan's pattern I'll say that this comment has done more to raise my confidence in SIAI than anything else in the recent context.

I'll solve your math problems if I can, I'll build Friendly AI for you if I can, if you think SIAI needs some kind of amazing PR person, give us enough money to hire one

I'm working on it, to within the limits of my own entrepreneurial ability and the costs of serving my own personal mission. Not that I would allocate such funds to a PR person. I would prefer to allocate it to research of the 'publish in traditional journals' kind. If I was in the business of giving advice I would give the same advice you have no doubt heard 1,000 times: the best thing that you personally could do for PR isn't to talk about SIAI but to get peer reviewed papers published. Even though academia is far from perfect, riddled with biases and perhaps inclined to have a certain resistance to your impingement it is still important.

or better yet, why don't you try being perfect and see whether it's as easy as it sounds while you're handing out advice?

Now, now, I think the 'give us the cash' helps you out rather a lot mor... (read more)

[-]Jordan14y140

Yes, the standards will keep going up.

And, if you draw closer to your goal, the standards you're held to will dwarf what you see here. You're trying to build god, for christ's sake. As the end game approaches more and more people are going to be taking that prospect more and more seriously, and there will be a shit storm. You'd better believe that every last word you've written is going to be analysed and misconstrued, often by people with much less sympathy than us here.

It's a lot of pressure. I don't envy you. All I can offer you is: =(

7multifoliaterose14y

Much of what you say here sounds quite reasonable. Since many great scientists have lacked social grace, it seems to me that your PR difficulties have no bearing on your ability to do valuable research. I think that the trouble arises from the fact that people (including myself) have taken you to be an official representative of SIAI. As long as SIAI makes it clear that your remarks do not reflect SIAI's positions, there should be no problem.

7wedrifid14y

There is an interesting question. Does it make a difference if one of the core subjects of research (rationality) strongly suggests different actions be taken and the core research goal (creating or influencing the creation of an FAI) requires particular standards in ethics and rationality? For FAI research behaviours that reflect ethically relevant decision making and rational thinking under pressure matter. If you do research into 'applied godhood' then you can be expected to be held to particularly high standards.

7multifoliaterose14y

Yes, these are good points. What I was saying above is that if Eliezer wants to defer to other SIAI staff then we should seek justification from them rather than from him. Maybe they have good reasons for thinking that it's a good idea for him to do FAI research despite the issue that you mention.

2wedrifid14y

I understand and I did vote your comment up. The point is relevant even if not absolutely so in this instance.

1LucasSloan14y

Is this now official LW slang?

8wedrifid14y

Only if we need official LW slang for "Have rigid boundaries and take decisive action in response to an irredeemable defection in a game of community involvement". I just mean to say it may be better to use a different slang term for "exit without leaving a trace" since for some "pull a Roko" would always prompt a feeling of regret. I wouldn't bring up Roko at all (particularly reference to the exit) because I want to leave the past in the past. I'm only commenting now because I don't want it to, as you say, become official LW slang.

0[anonymous]14y

I'm probably alone on this one, but "charming modesty" usually puts me off. It gives me the impression of someone slimy and manipulative, or of someone without the rigor of thought necessary to accomplish anything interesting.

[-]Jordan14y110

Well said.

In the past I've seen Eliezer respond to criticism very well. His responses seemed to be in good faith, even when abrasive. I use this signal as a heuristic for evaluating experts in fields I know little about. I'm versed in the area of existential risk reduction well enough not to need this heuristic, but I'm not versed in the area of the effectiveness of SIAI.

Eliezer's recent responses have reduced my faith in SIAI, which, after all, is rooted almost solely in my impression of its members. This is a double stroke: my faith in Eliezer himself is reduced, and the reason for this is a public appearance which will likely prevent others from supporting SIAI, which is more evidence to me that SIAI won't succeed.

SIAI (and Eliezer) still has a lot of credibility in my eyes, but I will be moving away from heuristics and looking for more concrete evidence as I debate whether to continue to be a supporter.

[-]Paul Crowley14y110

Tone matters, but would you really destroy the Earth because Eliezer was mean?

5wedrifid14y

The issue seems to be not whether Eliezer personally behaves in an unpleasant manner but rather how Eliezer's responses influence predictions of just how much difference Eliezer will be able to make on the problem at hand. The implied reasoning Jordan makes is different to the one you suggest.

[-]Eliezer Yudkowsky14y120

BTW, when people start saying, not, "You offended me, personally" but "I'm worried about how other people will react", I usually take that as a cue to give up.

6wedrifid14y

This too is not the key point in question. Your reaction here can be expected to correlate with other reactions you would make in situations not necessarily related to PR. In particular it says something about potential responses to ego damaging information. I am sure you can see how that would be relevant to estimates of possible value (positive or negative) that you will contribute. I again disclaim that 'influences' does not mean 'drastically influences'. Note that the reply in the parent does relate to other contexts somewhat more than this one.

1XiXiDu14y

Maybe because most people who talk to you know that they are not typical and also know that others wouldn't be as polite given your style of argumentation?

2Eneasz14y

People who know they aren't typical should probably realize that their simulation of typical minds will often be wrong. A corollary to Vinge's Law, perhaps?

6Paul Crowley14y

That's still hard to see - that you could go from thinking that SIAI represents the most efficient form of altruistic giving to that it no longer is, because Eliezer was rude to someone on LW. He's manifestly managed to persuade quite a few people of the importance of AI risk despite these disadvantages - I think you'd have to go a long way to build a case that SIAI is significantly more likely to founder and fail on the basis of that one comment.

5wedrifid14y

Rudeness isn't the issue* so much as what the rudeness is used instead of. A terrible response in a place where a good response should be possible is information that should have influence on evaluations. * Except, I suppose, PR implications of behaviour in public having some effect, but that isn't something that Jordan's statement needs to rely on. Obviously. But not every update needs to be an overwhelming one. Again, the argument you refute here is not one that Jordan made. EDIT: I just saw Jordan's reply and saw that he did mean both these points but that he also considered the part that I had included only as a footnote.

2Jordan14y

Yes, thank you for clarifying that.

7Eliezer Yudkowsky14y

Can you please be more precise about what you saw as the problem?

[-]Jordan14y300

Yes. Your first line

I have difficulty taking this seriously. Someone else can respond to it.

was devoid of explicit information. It was purely negative.

Implicitly, I assume you meant that existential risk reduction is so important that no other 'normal' charity can compare in cost effectiveness (utility bought per dollar). While I agree that existential risk reduction is insanely important, it doesn't follow that SIAI is a good charity to donate to. SIAI may actually be hurting the cause (in one way, by hurting public opinion), and this is one of multi's points. Your implicit statement seems to me to be a rebuke of this point sans evidence, amounting to simply saying nuh-uh.

You say

you don't make clear what constitutes a sufficient level of transparency and accountability

which is a good point. But then go on to say

though of course you will now carefully look over all of SIAI's activities directed at transparency and accountability, and decide that the needed level is somewhere above that.

essentially accusing multi of a crime in rationality before he commits it. On a site devoted to rationality that is a serious accusation. It's understood on this site that there are a mil... (read more)

[-]Eliezer Yudkowsky14y210

Thank you for your specifications, and I'll try to keep them in mind!

7multifoliaterose14y

I'll reply in more detail later but for now I'l just say This sounds great I'm not one of those people, I would be happy to donate to SIAI and encourage others to do so if I perceive a significant change from the status quo and there's no new superior alternative that emerges. I think that if SIAI were well constituted, donating to it would be much more cost effective than VillageReach. I would be thrilled to see the changes that I would like to see take place. More on precisely what I'm looking for to follow.

8Torben14y

For most realistic interest rates this statement would have made it more rational to put your previous traditional aid donation into a banking account for a year to see if your bet had come out -- and then donating to SIAI.

4multifoliaterose14y

I donate now using GiveWell to signal that I care about transparency and accountability to incentivize charities to improve.

[-]CarlShulman14y220

You would have done that by putting the money in a DAF and announcing your policy, rather than providing incentives in what you say is the wrong field. You're signaling that you will use money wastefully (in its direct effects) by your own standards rather than withhold it until a good enough recipient emerges by your standards.

8wedrifid14y

This is an effective signal. It sounds like multi considers changing the current standard of existential risks charities from ineffective to effective far more important than boosting the income of said class of charities. This being the case showing that there is a potential supply of free money for any existential risk focussed charity that meets the standards multi considers important. As well as influencing the world in the short term and satisfying his desire to signal and feel generous donating to Givewell effectively shows that multi is willing to put his money where is mouth is. He knows that if he wasn't currently donating Eliezer would use that as another excuse to avoid the issue, since Eliezer has previously declared that to be his policy.

4gwern14y

DAF?

8CarlShulman14y

Donor-Advised-Fund, a vehicle which allows one to place money into an investment account where it can only be used to make charitable donations. It allows you to commit yourself to giving (you can't take the money back to spend on beer) even if you don't know what's best now, or can allow you to accumulate donations over time so that you can get leverage with the simultaneous disbursal of a big chunk. Here's the Fidelity DAF site.

4wedrifid14y

I like the site! The limits may be a little frustrating for some potential users... you need to start with US$5,000. (And pay a minimum of $60 per year in fees).

0katydee14y

What exactly do you mean by leverage?

3CarlShulman14y

When you're giving a large donation all at once, the transaction costs of meeting your demands are smaller relative to the gains, and the transaction costs of doing nonstandard donations (e.g. teaming up with others to create a whole new program or organization) are more manageable.

5katydee14y

I don't get it, why would you want to make demands? Isn't the point of donating that you think others are better positioned to accomplish goal X than you are, so they're able to make more efficient use of the money?

2CarlShulman14y

E.g. demands for work to be done providing information to you, or to favor a specific project (although the latter is more murky with fungibility issues).

1juliawise13y

Charities sometimes favor the work they believe to be popular with donors over the work they believe would be more useful. Specifically, I'm thinking of monitoring and evaluation. By designating money for unpopular but useful tasks, you encourage them to better fund it. Before doing this, I would talk to the organizations you're considering funding and find out what unsexy projects they would like to fund more. Then decide if you think they're worth funding.

3utilitymonster14y

This doesn't sound like a bad idea. Could someone give reasons to think that donations to SIAI now would be better than this?

4CarlShulman14y

In your specific case, given what you have said about your epistemic state, I would think that you subjectively-ought to do something like this (a commitment mechanism, but not necessarily with a commitment to reducing existential risk given your normative uncertainty). I'll have more to say about the general analysis in 48 hours or more, following a long flight from Australia.

0[anonymous]14y

Does "this" mean DAF, or signalling through waste?

3Tyrrell_McAllister14y

One problem with this is that skeptics like Eliezer would assume that mulitfoliaterose will just move the goal posts when it comes time to pay.

3wedrifid14y

Fortunately a specific public commitment does at least help keep things grounded in more than rhetoric. Even assuming no money eventuates there is value in clearly meeting the challenge to the satisfaction of observers.

3Tyrrell_McAllister14y

Are you reading multifoliaterose carefully? He has made neither of these claims. He said that supporting a tuberculosis charity is better than donating to SIAI, not that supporting a tuberculosis charity is the best way to fight existential risk. And he hasn't advocated using something other than the expected case when evaluating a non-transparent charity. What you may infer is that he believes that the worst case does not significantly differ from the expected case in the context of the amount of money that he would donate. That belief may not be realistic, but it's not the belief that you impute to him.

9Airedale14y

I hesitate to point to language from an earlier version of the post, since multifoliaterose has taken out this language, but given that EY was responding to the earlier version, it seems fair. The original post included the following language: (emphasis added) I believe there originally may have been some links there, but I don't have them anymore. Nonetheless, if I correctly understand the references to utilitymonster, astronomical waste, and Shut Up and multiply, I do think that that multifoliaterose was arguing that even the sorts of donors most interested in minimizing existential risk should still give to those other charities. Does that reading seem wrong?

4Tyrrell_McAllister14y

Here is my reading: Even in the case of utilitymonster, * his/her concern about tuberculosis (say) in the near term is high enough, and * SIAI's chances of lowering existential risk by a sufficient amount are low enough, to imply that utilitymonster would get more expected utility from donating to StopTB than from donating to SIAI. Also, multi isn't denying that utilitymonster's money would be better spent in some third way that directly pertains to existential risk. (However, such a denial may be implicit in multi's own decision to give to GiveWell's charities, depending on why he does it.)

5Airedale14y

I don't know that we disagree very much, but I don’t want to lose sight of the original issue as to whether EY’s characterization accurately reflected what multifoliaterose was saying. I think we may agree that it takes an extra step in interpreting multifoliaterose’s post to get to EY’s characterization, and that there may be sufficient ambiguity in the original post such that not everyone would take that step: I did implicitly read such a denial into the original post. As Carl noted: For me, the references to the Givewell-approved charities and the lack of references to alternate existential risk reducing charities like FHI seemed to suggest that multifoliaterose was implicitly denying the existence of a third alternative. Perhaps EY read the post similarly.

1Tyrrell_McAllister14y

I agree that this is the most probable meaning. The only other relevant consideration I know of is multi's statement upstream that he uses GiveWell in part to encourage transparency in other charities. Maybe he sees this as a way to encourage existential-risk charities to do better, making them more likely to succeed.

2CarlShulman14y

Well, since multifoliaterose himself has been giving all of his charitable contributions to VillageReach, it's a sensible reading.

0[anonymous]14y

That's not what Multi said. He said we should assume the worst. You only need to assume something when you know that belief would be useful even though you don't believe it. So he clearly doesn't believe the worst (or if he does, he hasn't said so). He also said that he "believe[s] that reducing existential risk is ultimately more important than developing world aid." How do you go from there to supporting StopTB over SIAI, unless you believe the worst?

0Tyrrell_McAllister14y

I don't use the word "assume" in the way that you describe, and I would be surprised if multi were. Here I think we more-or-less agree. On my reading, multi is saying that, right now, the probability that SIAI is a money pit is high enough to outweigh the good that SIAI would do if it weren't a money pit, relative to a tuberculosis charity. But multi is also saying that this probability assignment is unstable, so that some reasonable amount of evidence would lead him to radically reassign his probabilities.

1XiXiDu14y

Transparency requires reevaluation if we are talking about a small group of people that are responsible to give the thing in control of the fate of the universe and all entities in it it's plan of action. That is, if you were able to evaluate that this is (1) necessary, (2) possible, (3) you are the right people for the job. This lack of transparency could make people think (1) this is bullshit, (2) it's impossible to do anyway, (3) you might apply a CEV of the SIAI rather than humanity, (4) given 3' the utility payoff is higher donating to GiveWell's top ranked charities.

0[anonymous]14y

I think that would be best, actually.

[-]PhilGoetz14y140

Your objections focus on EY. SIAI > EY. Therefore, SIAI's transparency <> EY's transparency. SIAI's openness to criticism > EY's openness to criticism.

Go to SIAI's website, and you can find a list of their recent accomplishments. You can also get a detailed breakdown of funding by grant, and make an earmarked donation to a specific grant. That's pretty transparent.

OTOH, EY is non-transparent. Deliberately so; and he appears to intend to continue to be so. And you can't find a breakdown on SIAI's website of what fraction of their donations go into the "Eliezer Yudkowsky black ops" part of their budget. It would be nice to know that.

If you're worried about what EY is doing and don't want to give him money, you can earmark money for other purposes. Of course, money is fungible, so that capability has limited value.

1multifoliaterose14y

Phil, I like your comments. Some points in response: •My concern is not really that the fraction of SIAI donations which go to EY is too large, but rather the (significant) possibility that SIAI is attracting donations and volunteers on the premise that EY's unjustified claims about the state of the world are true. I think that it's very important that SIAI make it clear precisely how much it claims to be true. •A large part of the issue is that there's ambiguity as to whether EY is properly regarded as a spokesperson for SIAI. As Airedale says As I said in response to EY here, as long as SIAI makes it clear that EY is not an official representative of SIAI and points to official representatives of SIAI who accurately reflect SIAI's position, there shouldn't be a problem.

[-]timtyler14y90

This recentish page was a step towards transparency:

http://singinst.org/grants/challenge

Transparency is generally desirable - but it has some costs.

6magfrump14y

I will donate $100 to SIAI after the publication of this paper.

2wedrifid14y

To make things easier (and more enjoyable) for you to follow through with this and also to make the words 'real' for observers and the SIAI would you consider using some form of escrow service? Pre-commitments are fun.

2magfrump14y

Yes, assuming it isn't difficult, and that I can delay until the school year starts (not until September 23rd for me) and I have a reliable income.

2jsalvatier14y

I would do this. I think GiveWell had a similar kind of setup at one point, having people commit to donating to their recommendation.

6multifoliaterose14y

Yes, I approve of the page that you link :-). The point of my post is to argue in favor of the idea that the costs are worth it.

[-]Eliezer Yudkowsky14y100

How much will you donate to cover the costs? It's always easy to spend other people's money.

[-]multifoliaterose14y150

Okay, thinking it over for the last hour, I now have a concrete statement to make about my willingness to donate to SIAI. I promise to donate $2000 to SIAI in a year's time if by that time SIAI has secured a 2-star rating from GiveWell for donors who are interested in existential risk.

I will urge GiveWell to evaluate existential risk charities with a view toward making this condition a fair one. If after year's time GiveWell has not yet evaluated SIAI, my offer will still stand.

[Edit: Slightly rephrased, removed condition involving quotes which had been taken out of context.]

[-]Jasen14y260

Jonah,

Thanks for expressing an interest in donating to SIAI.

(a) SIAI has secured a 2 star donation from GiveWell for donors who are interested in existential risk.

We think that UFAI is the largest known existential risk and that the most complete solution - FAI - addresses all othe... (read more)

[-]multifoliaterose14y120

If GiveWell decides not to endorse charities focused on existential risk reduction as a general policy, there is little we can do about it. Would you consider an alternative set of criteria if this turns out to be the case?

Yes, I would consider an alternative set of criteria if this turns out to be the case.

I have long felt that GiveWell places too much emphasis on demonstrated impact and believe that in doing so GiveWell may be missing some of the highest expected value opportunities for donors.

It would have been a good idea for you to watch the videos yourself before assuming that XiXiDu's summaries (not actual quotes, despite the quotation marks that surrounded them) were accurate.

I was not sure that XiXiDu's summaries were accurate which is why I added a disclaimer to my original remark. I have edited my original comment accordingly.

I apologize to Eliezer for inadvertantly publicizing a misinterpretation of his views.

[-]jimrandomh14y140

(b) You publically apologize for and qualify your statements quoted by XiXiDu here. I believe that these statements are very bad for public relations. Even if true, they are only true at the margin and so at very least need to be qualified in that way.

(Note: I have not had the chance to verify that XiXiDu is quoting you correctly because I have not had access to online video for the past few weeks - condition (b) is given on the assumption that the quotes are accurate)

It is always wrong to demand the retraction of a quote which you have not seen in context.

6multifoliaterose14y

Thanks, I have edited my comment accordingly.

2multifoliaterose14y

As I said, condition (b) is given based on an assumption which may be wrong. In any case, for public relations purposes, I want SIAI to very clearly indicate that it does not request arbitrarily large amounts of funding from donors.

7JGWeissman14y

In one of the videos XiXiDu cites as reference, Eliezer predicts that funding at the level of a billion dollars would be counterproductive, because it would attract the wrong kind of attention.

2multifoliaterose14y

Thanks for pointing this out. I edited my comment accordingly.

4timtyler14y

Statements misquoted by XiXiDu - it looks like!

3multifoliaterose14y

Thanks for pointing this out

4multifoliaterose14y

Sorry for attaching a misrepresentation of your view to you based on descriptions of videos which I had not seen! I have edited my other comment accordingly.

2multifoliaterose14y

I donated 10% of my annual (graduate student) income to VillageReach a few weeks ago. As I say in my post, this is not because I have special attachment to international aid. I would be willing to donate to SIAI if SIAI coud convince me that it would use the money well. At present I don't even know how much money SIAI receives in donations a year, much less how it's used and whether the organization has room for more funding. I believe that there are others like me and that in the long run exhibiting transparency would allow SIAI to attract more than enough extra donations to cover the costs of transparency. Note that GiveWell leveraged 1 million dollars last year and that this amount may be increasing exponentially (as GiveWell is still quite young).

4Liron14y

I would also like to see SIAI post a description of its finances and leadership structure.

6Airedale14y

I agree it would be good if more info on finances was readily available. There are tax returns (although I think the most recent is 2008) available on Guidestar (with free registration). But as for leadership structure, is this link the sort of thing you had in mind or were you looking for an actual org chart or something?

7Morendil14y

Having run a small non-profit operation for a few years now, the standard of transparency I now like is simply publishing our General Ledger to the Web every year. What's nice about it: it's a) feasible (our accounts are in Xero and once you've figured out the export it's a breeze), b) the ultimate in transparency. We still do summary reports to give people an idea of what's happened with the money, but anyone who complains or for some other reason wants the raw data, I can just point at the GL.

1Liron14y

OK, the leadership structure info is satisfactory.

[-]XiXiDu14y60

I've suspected for a long time that the movement around EY might be a sophisticated scam to live off donations of nonconformists...

Before someone brings it up. I further said, "but that's just another crazy conspiracy theory". I wouldn't say something like this and actually mean it without being reasonable confident. And I'm not, I'm just being a troll sometimes. Although other people will inevitably come to this conclusion. You just have to read this post, see the massive amount of writings about rationality and the little cherry on the cake ... (read more)

[-]pjeby14y140

They first need to figure out how to distill their subject matter and make it more palatable for the average person who really doesn't care about being part of the Bayesian in-crowd.

There's always HP:MOR. ;-)

2wedrifid14y

Please don't generalise. I certainly can't be accused of thinking we succeed in supplying an adequate response. It's not you against lesswrong. It is you against "Oh $#!%, oh $#!%, we're all going to die! What can we do to survive and thrive?"

[-]timtyler14y50

I must say that, in fact, much of the nonprofit sector fits incredibly better into Prof. Hanson’s view of charity as “wasteful signaling” than into the traditional view of charity as helping.

Charity is largely about signalling. Often, signalling has to be expensive to be credible. The "traditional view of charity" seems like naive stupidity. That is not necessarily something to lament. It is great that people want to signal their goodness, wealth, and other virtues via making the world a better place! Surely it is best if everyone involved understands what is going on in this area - rather than living in denial of human nature.

1zero_call14y

I would argue that charity is just plain good, and you don't need to take something simple and kind and turn it into an inconclusive exercise in societal interpretation.

4Torben14y

Are you familiar with the Hansonian view of signaling?

[-]Larks14y30

SIAI does not presently exhibit high levels of transparency and accountability... For this reason together with the concerns which I express about Existential Risk and Public Relations, I believe that at present GiveWell's top ranked charities VillageReach and StopTB are better choices than SIAI

Suppose SIAI were a thousand times less accountable than VillageReach.
Suppose this made SIAI a million times less effective than it could be.
Suppose that even the most efficient Existential Risk charity could only reduce P(uFAI|AGI) by 10^-9
Suppose the odds o

... (read more)

[-]rwallace14y190

A problem with Pascal's Mugging arguments is that once you commit yourself to taking seriously very unlikely events (because they are multiplied by huge potential utilities), if you want to be consistent, you must take into account all potentially relevant unlikely events, not just the ones that point in your desired direction.

To be sure, you can come up with a story in which SIAI with probability epsilon makes a key positive difference, for bignum expected lives saved. But by the same token you can come up with stories in which SIAI with probability epsilon makes a key negative difference (e.g. by convincing people to abandon fruitful lines of research for fruitless ones), for bignum expected lives lost. Similarly, you can come up with stories in which even a small amount of resources spent elsewhere, with probability epsilon makes a key positive difference (e.g. a child saved from death by potentially curable disease, may grow up to make a critical scientific breakthrough or play a role in preserving world peace), for bignum expected lives saved.

Intuition would have us reject Pascal's Mugging, but when you think it through in full detail, the logical conclusion is that we should... reject Pascal's Mugging. It does actually reduce to normality.

[-]cousin_it14y100

Wow. Do you really think this sort of agrument can turn people to SIAI, rather than against any cause that uses tiny probabilities of vast utilities to justify itself?

8Nick_Tarleton14y

Seconded, tentatively. I'm afraid that all arguments of this form (for x-risk reduction, Pascal's Wager, etc.) can't avoid being rejected out of hand by most people, due to their insatiability: buying even a little bit into such an argument seems to compel a person to spend arbitrary amounts of resources on the improbable thing, and open them to harsh criticism for holding back even a little. That said, a few sincere consequentialists actually will react positively to such arguments, so maybe making them on LW is worthwhile on balance.

3Larks14y

Multifoliaterose said his result held even for donors who took Astronomical Waste seriously. This seems unlikely to be the case. Edit: I didn't vote you down, but what? SIAI is an Existential Risk charity: the point is to save the entire human race, at low probability. Of course the expected value is going to be calculated by multiplying a tiny probability by an enormous value!

2cousin_it14y

This isn't true for all existential risks. For example, fears about nuclear war or global warming don't rely on such tiny probabilities. But discussions of many other risks remind me of this xkcd.

2CarlShulman14y

The usual consequentialist case for charity to reduce risks of nuclear war or catastrophic global warming feedbacks does rely on tiny probabilities of donations making a difference. Likwise for voting and for many kinds of scientific and medical research charity. Edit: not as tiny as in Lark's comment, although the numbers there are incredibly low.

2steven046114y

If the Pascal's mugging issue is with exerting a small force on a big thing and therefore having a small probability of succeeding, I don't think that's even a coherent objection; in a chaotic universe, anything you do may save or destroy individual people and human civilization by redrawing from the distribution, and shifting the distribution of the number of saved lives up by one through e.g. aid or buying someone cryo doesn't seem fundamentally different from e.g. shifting the distribution of election outcomes. This is more clearly true if you believe in a multiverse such as MWI, but I don't think it requires that. ETA: if your distribution on people killed is uniform for say 6 through 10, then definitely saving one person and having a 1 in 5 chance of turning a 10-death situation into a 5-death situation are the same thing except for maybe the identity of the victims, right?

-1cousin_it14y

That's a very different value of "tiny" than the one in Larks's comment.

5timtyler14y

This is just Pascal's mugging again. If you get to make up the numbers then you can make things turn out however you like.

3Larks14y

Not if you deliberately pick numbers orders of magnitude more unfavourable to your point of view than you think is actually the case.

4Perplexed14y

But you didn't do that. A calculation involves both quantitative and qualitative/structural assumptions, and you slanted all of your qualitative choices toward making your point. You used a discount rate of 0. (That is, a hypothetical life a hundred generations from now deserves exactly as much of my consideration as people alive today). That totally discredits your calculation. You used a definition of 'saving a life' which suggests that we ought to pay women to get pregnant, have children, and then kill the infant so they are free to get pregnant again. Count that as one life 'saved'. You didn't provide estimates of the opportunity cost in 'human lives' of diverting resources toward colonizing the universe. Seems to me those costs could be enormous - in lives at the time of diversion, not in distant-future lives.

4Larks14y

Appart from the issue JGWeissman brings up, even if you supposed that the lives saved all occurred 300 years in the future, a reasonable discount rate would still only give you a couple of orders of magnitude. For example, 1.05^300 = 2.3 * 10^6 Which is nowhere near enough. Edit: there are discount rates that would give you the result you want, but it still seems pritty plausible that, assuming Astronomical Waste, SIAI isn't a bad bet.

2Perplexed14y

Sounds about right to me. Huh? You think 6 is "a couple"? I wish I had your sex life! But 5% per annum is far too high. It discounts the next generation to only a quarter of the current generation. Way too steep. Double huh? And huh? some more. You wrote: You imagine (conservatively) that there are a potential 10^18 lives to save 300 years into the future? Boy, I really wish I had your sex life.

1CarlShulman14y

If people 300 years from now are whole brain emulations or AIs, then they could reproduce like software with high population densities.

1timtyler14y

Alternatively, if the human-size brains were all sucked into the matrix long ago, there may well be about 1 person per-planet.

1JGWeissman14y

What makes a life at one time worth more than a life at a different time?

2Perplexed14y

1. Distance. 2. Tradition in the field of economics 3. Mathematical well-behavedness may demand this if the universal expansion is not slowing down. 4. Reciprocity. Future folks aren't concerned about my wishes, so why should I be concerned about theirs? 5. What makes a life at one time worth the same as a life at a different time? In a sense, these are flip answers, because I am not really a utilitarian to begin with. And my rejection of utilitarianism starts by asking how it is possible to sum up utilities for different people. It is adding apples and oranges. There is no natural exchange rate. Utilities are like subjective probabilities of different people - it might make sense to compute a weighted average, but how do you justify your weighting scheme? I suspect that discussing this topic carefully would take too much of my time from other responsibilities, but I hope this sketch has at least given you some things to think about.

5Tyrrell_McAllister14y

Utilities for different people don't come into it. The question is, how much to you now is a contemporaneous person worth, versus someone in a hundred generations? (Or did you mean utilities of different people?)

1Perplexed14y

You have me confused now. I like apples. Joe likes oranges. Mary wishes to maximize total utility. Should she plant an orange tree or an apple tree? I agree with Kant (perhaps for the only time). Asking how much Joe is worth to Mary is verbotten.

6Tyrrell_McAllister14y

She should consider the world as it would be if she planted the apple tree, and the world as it would be if she planted the orange tree, and see which one is better as she measures value. (Another option is to plant neither, of course.) The idea isn't to add up and maximize everyone's utility. I agree with you that that makes no sense. The point is, when an agent makes a decision, that agent has to evaluate alternatives, and those alternatives are going to be weighed according to how they score under the agent's utility function. But utility isn't just selfish profit. I can value that there be happiness even if I don't ever get to know about it. 1. We don't always have the luxury of not choosing. What should Mary do in a trolley problem where she has to direct a train at either you or Joe (or else you both die)? That said . . . 2. "Worth" needs to be understood in an expansive sense. Kant is probably right that Mary shouldn't think, "Now, from whom can I extract more profit for my selfish ends? That's the one I'll save." The things she ought to consider are probably the ones we're accustomed to thinking of as "selfless". But she can't evade making a decision.

0Perplexed14y

Ok, I think we can agree to agree. Revealed preference doesn't prevent me from incorporating utilitarianish snippets of other peoples utility judgments into my own preferences. I am allowed to be benevolent. But simple math and logic prevent me from doing it all-out, the way that Bentham suggested. Now, which one of us has to tell Eliezer?

1timtyler14y

What simple math and logic? Utilitarianism seems pretty silly to me too - but adding different people's utilities together is hardly a show-stopping problem. The problem I see with utilitarianism is that it is a distant ideal. Ideals of moral behaviour normally work best when they act like a carrot which is slightly out of reach. Utilitarianism conflicts with people's basic drives. It turns everyone into a sinner. If you preach utilitarianism, people just think you are trying to manipulate them into giving away all their stuff. Usually that is true - promoters of utilitarianism are usually poorer folk who are after the rich people's stuff - and have found a moral philosophy that helps them get at it. Politicians often say they will tax the rich and give the money to the poor. This is because they want the poor people's votes. Utilitarianism is the ethical equivalent of that. Leaders sometimes publicly promote such policies if they want the support of the masses in order to gain power.

2JGWeissman14y

No, what is forbidden is to ask how much Joe is worth in an absolute sense, independent of an agent like Mary. Utility is not a fundamental property of the world, it is perceived by agents with preferences.

1Perplexed14y

This is rapidly becoming surreal. Forbidden is not a fundamental property of the world, it is imposed by theorists with agendas. Samuelson, Wald, von Neumann, Savage, and the other founders of "revealed preference" forbid us to ask how much Joe (or anything else) is worth, independent of an agent with preferences, such as Mary. Emmanuel Kant, and anyone else who takes "The categorical imperative" at all seriously, forbids us to ask what Joe is worth to Mary, though we may ask what Joe's cat Maru is worth to Mary. I knew I shouldn't have gotten involved in this thread.

5timtyler14y

Immanuel Kant says we that can't ask what Joe is worth to Mary? So what? Why should anyone heed that advice? It is silly.

-1timtyler14y

Utilities of different people, yes. He's complaining that they don't add up.

0timtyler14y

For 2, perhaps consider: http://lesswrong.com/lw/n2/against_discount_rates/

1Perplexed14y

Considered. Not convinced. If that was intended as an argument, then EY was having a very bad day. He is welcome to his opinion but he is not welcome to substitute his for mine. The ending was particularly bizarre. It sounded like he was saying that treasury bills don't pay enough interest to make up for the risk that the US may not be here 300 years from now. But we should, for example, consider the projected enjoyment of people we imagine visiting our nature preserves 500 years from now, as if their enjoyment were as important as our own, not discounting at all for the risk that they may not even exist.

4Nick_Tarleton14y

Eliezer doesn't disagree: as he says more than once, he's talking about pure preferences, intrinsic values. Other risks do need to be incorporated, but it seems better to do so directly, rather than through a discounting heuristic. Larks seems to implicitly be doing this with his P(AGI) = 10^-9.

1timtyler14y

Time travel, the past "still existing" - and utilitariainism? I don't buy any of that either - but in the context of artificial intelligence, I do agree that building discounting functions into the agent's ultimate values looks like bad news. Discounting functions arise because agents don't know about the future - and can't predict or control it very well. However, the extent to which they can't predict or control it is a function of the circumstances and their own capabilities. If you wire temporal discounting into the ultimate preferences of super-Deep Blue - then it can't ever self-improve to push its prediction horizon further out as it gets more computing power! You are unnecessarily building limitations into it. Better to have no temporal discounting wired in - and let the machine itself figure out to what extent it can predict and control the future - and so figure out the relative value of the present.

2timtyler14y

Well, in that case, your math is all wrong! The SIAI is actually having a negative effect, by distracting conscientious researchers with all kinds of ridiculous and irrelevant moral issues - which results in the unscrupulous Korean FKU corporation developing machine intelligence first - when otherwise they would have been beaten to it. Their bot promises humans a cure for cancer - but then it gets powerful, eats the internet, and goes on to do all manner of dubious things. If only you had had the sense to give to VillageReach!

4CarlShulman14y

With such low numbers the incidental effects of VillageReach on existential risk (via poverty, population, favoring the efficient philanthropy culture, etc) would dominate. If reasonably favorable, they would mean VillageReach would be a better choice. However, the numbers in the comment are incredibly low.

3ata14y

We need to be careful with the terminology here. "Saving expected lives" and "expecting to save lives" are pretty different activities.

0Larks14y

Thanks & corrected.

0EStokes14y

Related: How Much it Matters to Know What Matters: A Back of the Envelope Calculation

[-]allenwang14y00

I really want to put up a post that is highly relevant to this topic. I've been working with a couple of friends on an idea to altar personal incentives to solve the kinds of public good provision problems that charities and other organizations face, and I want to get some feedback from this community. Is someone with enough points to post willing to read over it and post for me? Or can I get some upvotes? (I know that this might be a bit rude, but I really want to get this out there ASAP).

Thanks a bunch, Allen Wang

0katydee14y

I can also take a look if you want, but yeah, no promises.

0whpearson14y

I'll have a look, no promises.

Moderation Log