Once there are a few thousand people working on existential risks, the marginal expected utility of recruiting another worker goes down. People start working at cross-purposes because of not knowing enough about each other's plans.

Rather than increasing the number of e-risk workers as fast as possible, the recruiting strategy that minimizes e-risks is to figure out what personal qualities make for the best e-risk workers and differentially to recruit people with those qualities.

And the most decisive personal quality I know about has to do with the "motivational structure" of the prospective worker. what natural human desires and pleasures (and perhaps natural human fears) motivate him or her? Two natural human motivations that cause many of the people currently working on e-risks or currently watching the public discourse about e-risks to persist in these activities are self-interest and altruism.

A lot of recruiting consists of written and oral communications. it is fairly straightforward to tailor the communications in such a way that it is of strong interest to, e.g., people interested in being altruistic while being boring to people motivated by, e.g., their own personal survival. It gets harder the more the reader knows about e-risks and about the singularity, but at present, most very bright people do not know much about these topics.

Consequently, communications that inform people about the singularity should be tailored to be interesting to those with the right motivations.

Since not enough people of sufficiently-high prestige advocate cryonics to make an argument for cryonics by authority persuasive, the only effective way to persuade people to sign up is with an argument on the scientific merits, which entails explaining to them about the singularity. I.e., communications whose purpose is to get people to sign up for cryonics is necessarily also communications that inform people about the singularity -- and this might be its more important effect even if the intent of the author is simply to get people to sign up for cryonics.

Maybe I'm wrong, but I tend to think that any discussion of cryonics I can imagine which is effective at getting people to sign up will tend to recruit the wrong people into the ranks of the singularitarians.

I am struck particularly at the difficultly of getting people to read communications about cryonics without appeal to their interest in their own survival.

And I think it increases existential risk to create communications that are informative about the singularity whose appeal is to the reader's survival interest.

People whose motivation is self-interest have a track record of being easy to control or influence by threats of imprisonment, violence, or other personal hardships. Scientific workers in particular have a track record of being easily cowed by bullies, e.g., government officials and business tycoons. Moreover, the prospective bullies tend to have goals that most people reading this would disapprove of.

There have probably not yet been any instances in which e-risks workers have been influenced or subverted by bullies, but there's a significant chance of it happening during the second phase of the work (when it will become obvious to any intelligent observer that the singularitarians command significant scientific and technical resources) if a significant fraction of singularitarians will have self-interested as their strongest motivation for persisting in their work towards the singularity.

I take my personal history as another piece of evidence for the hypothesis that people should discourage knowledge of the singularity and work on the singularity by people whose motivation for doing so is self-interest. When I was growing up in the 1970s, one of the books in my house, Get Ready for Immortality by Lawrence Lamb, M.D., caused me come to hope to live millions of years. It was not a major theme in my thinking, but it was definitely there. And I do not consider the person I was in my teens (or even my early 20s) to be the sort of person that should be encouraged to learn about and work toward the singularity.

By the time I came upon Eliezer's writings when I was 40 or 41, I had lost any desire to live millions of years. My motivation for trying to understand Eliezer's point about seed AI was to help me predict whether the research I was doing on new programming languages might have far-reaching negative consequences (by giving AGI researchers tools they might otherwise not come to have). (If they did have negative consequences, I meant to leave them unpublished.) I.e., my motivation was to avoid doing harm by not thinking through the consequences of some research I had done for the pleasure of scientific discovery.

And that is an example of the kind of motivation that should IMHO be encouraged. And note that it is quite easy to write in such a way that is boring to the hypothetical reader who is motivated only by maximizing the odds of his or her own survival and interesting to the hypothetical reader who is interested only in learning enough about the likely consequences of his or her behavior (and the behavior of his or her friends and coworkers) to make sure he or she is not unknowingly doing harm.

So, that is a little bit about why I tend not to think that persuading people to sign up for cryonics reduces existential risk. Yes, I see the same positive effects as JGWeissman, angry parsley and Eliezer. But I estimate the negative effects I have just described to outweigh them.

Getting people to sign up for cryonics is certainly not a strategy I would choose if my gaol was to minimize existential risks. (I.e., there are almost certainly better strategies.)

Well, what's my personal motivation, then, if I am engaging in motivated cognition?

But I do concede that my comment has a big problem here: "provides a personal incentive to hurry the intelligence explosion along so that it occurs before the death of the people signed up for cryonics" and I would have deleted my comment had you not replied already. Give me a few minutes to try to reconstruct the thinking that led to my conclusion.

One part is that getting people who are living now to hope to live a very long time disincentivizes them to consider strategies in which the singularity happens after they die.

But there was another part ISTR.

cryonics has the advantage of forcing you to care about the future. It provides an incentive to donate to fighting existential risk.

It also provides a personal incentive to hurry the intelligence explosion along so that it occurs before the death of the people signed up for cryonics. [ADDED: I concede that what I just said does not make sense; I went to delete it a few minutes after I submitted it but people had already replied. Please do not reply to this.] In other words, it provides a disincentive to pursue a strategy that discourages or suppresses existentially-risky research (on, e.g., AGI) so that less-risky research represents a larger share of the total research. In other words, it recruits the people most able to understand and to respond effectively to existential risks to spend (collectively) many millions of dollars in such a way that gives them a personal disincentive to pursue what I consider a very worthwhile strategy for addressing existential risks posed by certain lines of scientific research.

Most people who have expressed an opinion seem to believe that there is no stopping or slowing down significantly lines of research that (like AGI) can be continued with just a PC and access to the open scientific literature. But I tend to think it can be stopped or slowed down a great deal if effective people put as much effort into explaining why it is bad as Eliezer and his followers are putting into convincing people to sign up for cryonics.

According to my models, convincing people to sign up for cryonics at the current time does nothing to reduce existential risk. The opposite, in fact.

When I'm hungry I eat, but then I don't go on eating some more just to maximize a function. Eating isn't something I want a lot of. Likewise I don't want a ton of survival, just a bounded amount every day.

It is important to note that survival can be treated as a "big goal". For example Hopefully Anonymous treats it that way: if the probability that the pattern that is "him" will survive for the next billion years were .999999, he would strive to increase it to .9999995.

Parenthetically, although no current human being can hold such a belief with such a high level of confidence, that does not mean Hopefully Anonymous's goal is undefined or would become undefined when his survival is assured at a sufficiently high probability: it just means that a subgoal of his goal is the coming into existence of an agent that can hold such beliefs with such a high level of confidence. (The most likely way that that would happen involves a greater-than-human intelligence's having the same goal as Hopefully Anonymous or having a strongly-related goal like giving all 6 billion "founding humans" whatever they want.)

Do people get their downvotes back when a downvoted comment is deleted?

As of about 9 months ago, no.

Hi Randall Randall! If you and I were to arrange to meet the first Wednesday after the holidays, I would put a few words about the meeting in my calendar. Do you really want my calendar to be cluttered up with "nice to have" and "we should re-read this sometime" reminders? Aren't you a little worried that the reminder of our meeting will get lost in the noise, and I will neglect to go?

And if you aren't worried because you know that if I do not show then I'll never see my leather jacket again, or whatever, then I would be worried. And I do not like to worry: I want to know that whatever reminder I put in my calendar will definitely be seen by me on the day I intended for me to see it. And that means never putting so many things on my calendar that I start to regret consulting it. In fact, sometimes a whole month goes by in which there are zero words and zero marks in my calendar for the whole month (but then I lead an unusually unstructured life).

That file named July 2010? Well, first, there is no file named June 2010 or May 2010: there is only one for every 3 months. Moreover, past experience suggests that I probably will not get around to looking in there till September or December.

That's actually not a bad idea. I think I'll adopt it. Easy enough to do with iCal or Google Calendar.

You might find it informative to know that personal-productivity guru David Allen advises against putting on one's calendar anything that will not "die" if you do not get to it that day.

The note I made went into a file that I will review July 2010. (I make such a file for every 3-month period coming up.)

Thanks for writing this, Eliezer.

I consider it so informative I wrote a note to myself to re-read it in 6 months.

The internet makes makes it very easy to switch to a different site or project. This possibility forms the "checks and balances" that are historically necessary for national governments to be moderately benevolent.

Yay checks and balances!

The conventional term for what you are describing is low exit costs.

And yes, exit costs are much lower for a participant in a group blog than they are for a citizen of a country, which greatly reduces the need for formal checks and balances.

Exit costs for example were atypically low for citizens of the early U.S. when there was still a Western frontier. Once the frontier closed (and once agriculture, fur trapping and mining no longer provided a good living relative to other occupations for a large fraction of the U.S. population) the formerly very libertarian U.S. became steadily more socialistic.

The governance of Wikipedia, for example, would probably be improved in my humble opinion if they did away with the elections to the board of directors and made it as easy as possible for individuals and groups to re-use Wikipedia's content in competing encyclopedias.

What Wikipedia does now (or rather did several years ago the last time I checked) is to make publically available a snapshot of Wikipedia made every few months. If re-use were made as easy as posible, there would be an API that competitors could subscribe to to get real-time notification every time an entry in Wikipedia changes. Although Wikipedia probably still has a recent changes page, the way it is now, if a competitor tried to scrape it, Wikipedia's administration would probably block their IP address, which of course increases exit costs. And alternative way for Wikipedia's administration to lower exit costs would be a mechanism by which prospective editors who disagree with Wikipedia's policies and editorial decisions can publish on Wikipedia's servers their own version of (some) Wikipedia entries and by which users could indicate a preference for getting the alternative version instead of the Wikipedia version when an alternative version is available.

This isn't a "puzzle". To some extent, it's a sanity check I'm performing on myself. [from parent]

I believe I know "the answer" with high probability, and want to see what others think [from great grandparent]

Well, I wish your original post had been a little clearer about your reason for posting. I had somehow gotten the idea that my doing the exercise would be a good way for me to improve my own rationality or the rationality of a bunch of other readers, not just helping you assess your own rationality. Asking everyone here to do your exercise is an expensive way to do the latter.

Reading about a current trial and its aftermath is a very inefficient way for the reader to improve his or her own rationality; would you not agree? Do you think that it is an efficient way for the reader to assess his or her own rationality? (I do not unless there is unusually strong evidence, e.g., from DNA, that is held back till everyone has given their probabilities.)

If I did not regard you as a positive contributor to LW, komponisto, I would not bother making these comments on your post.

But, OK, now that you have my attention, I will respond to your human need: yeah, it is shocking how badly judicial systems can malfunction and how bad most people (even judges) are at rationality. Probably the thing that has most helped me tolerate these shocks is caring personal relationships with people whose rationality is above average. There is something about really being heard and understood by another human being about some point that will trip up many people that makes this sort of thing more tolerable.