Wow. Coordination is hard ;)

It is possible that this is just a phase I am going through, but if it is, it is a long phase.

The selection filter does not have to be total or near total for my point to stand, namely, Rule-of-Succession-like calculations can be useful even when one has enough information to think about the selection effects involved (provided that Rule-of-Succession-like calculations are ever useful).

And parenthetically selection effects on observations about whether nuclear exchanges happened in the past can be very strong. Consider for example a family who has lived in Washington, D.C., for the last 5 decades: Washington, D.C., is such an important target that it is unlikely the family would have survived the launch of most or all of the Soviet/Russian arsenal at the U.S. So, although I agree with you that the human race as a whole would probably have survived almost any plausible nuclear exchange, that does not do the family in D.C. much good. More precisely, it does not do much good for the family's ability to use historical data on whether or not nukes were launched at the U.S. in the past to refine their probability of launches in the future.

Regarding your second paragraph, Sir Gwern, if we switch the example to the question of whether the US and Russia will launch nukes at each other this year, I have at lot of information about the strength of the selection bias (including for example Carl Sagan's work on nuclear winter) that I might put to good use if I knew how to account for selection effects, but I would be sorely tempted to use something like the Rule of Succession (modified to account for the selection bias and where the analog of a day in which the sun might or might not rise is the start of the part of the career of someone in the military or in politics during which he or she can influence whether or not an attempt at a first strike is made) because my causal model of the mental processes behind the decision to launch is so unsatisfactory.

This might be a good place for me to point out that I never bought into the common wisdom, which I have never seen anyone object to or distance themselves from in print, that the chances of a nuclear exchange between the US and Russia went down considerably after the collapse of the Soviet Union in 1991.

If you know you probably would not have survived the sun's having failed to rise, you cannot just apply the Rule of Succession to your knowledge of past sunrises to calculate the probability that the sun will rise tomorrow because that would be ignoring relevant information, namely the existence of a severe selection bias. (Sadly, I do not know how to modify the Rule of Succession to account for the selection bias.)

When you say "more attractive", do you mean more attractive to people who have not gambled yet or more habit-forming?

What is the feature that makes it so deadly? I suggest that it is the random reinforcement schedule: Every five minutes you "press the lever", that is, check forum X or site Y. And every six or seven checks you get the reward: Someone posted something interesting!

What I want to know is whether a random reinforcement schedule becomes more addictive when we add the possibility that pressing the lever will lead to something aversive (e.g., a particularly stupid comment or one in which someone expresses enthusiasm for a course of action I think will cause more harm that good). I kind of suspect that such a schedule is better at hooking me than a random reinforcement schedule without the possibility of aversive outcome.

ADDED. For example, I note that reading internet forums open to all comers like LW causes aversive reactions in me much more frequently than does reading professionally authored and edited publications, e.g., The Atlantic or Smithsonian magazine, and I note that I get more hooked on the former than on the latter.

When I put on my donor hat, that is, when I imagine my becoming a significant donor, I tend in my imaginings and my plans to avoid anything that interferes with deriving warm fuzzies from the process of donating or planning to donate -- because when we say "warm fuzzies" we are referring to (a kind of) pleasure, and pleasure is the "gasoline" of the mind: it is certainly not the only thing that can "power" or "motivate" mental work, but it is IMHO the best fuel for work that needs to be sustained over a span of years. (And, yes, that is probably an argument against "Purchasing Fuzzies and Utilons Separately" in some situations although I did not have time today to re-read that article to see whether it can be reconciled with this comment.)

And, yeah, seeing money I donate (or simply imagining the money I will donate in the future) go to improving the lives of people who are probably not much better than me, but who spent a big fraction of their time and energy competing for status within the singularitarian community, jobs and donations with the likes of me, is one of the things that would probably interfere with my deriving warm fuzzies from the whole years-long and hopefully decades-long long process of my becoming a significant donor.

Certainly I am not alone in this aspect of my psychology. Now I will grant that a philanthropist can get a lot of donations by ignoring people who react like I do (namely, react with resentment) to high levels of prestige-seeking and impression management. But I tend to believe that to a philanthropist, donors are like customers are to a consultancy or investors are to a fast-growing company: the quality of the thinking of one's donors (and in particular whether those donors got into donating out of a subconscious desire to affiliate with high-status folk) will tend to have a large effect on one's sanity and ability to reach one's goals.

And let me stress again that at present the level of prestige-seeking and impression management by insiders at SI is low enough not to cause my resentment to build up to levels that would cause me to start thinking about directing my donations elsewhere. But that might change if enough people with Holden-Karnofsky levels of credibility and influence exhort SI to increase their levels of prestige-seeking and impression management.

ADDED. The thing that is wrong with this comment and probably some of my other comments in this thread is that some of my remarks seem to be addressed to people seeking donations. If I were a better communicator, I would have made it clear that the target audience for my comments is donors. I am not worried about persuading people seeking donations because I am confident that if there were some barrier to my donating to, e.g., SI and FHI, I will manage to find other ways of purchasing utilons of comparable or almost-comparable efficiency.

One last thing I would say to donors and wanna-be donors is that this tendency towards resentment I have been describing in this comment (and the resulting inhibitory effect on my motivation) can be considered a feature (rather than a bug) of my personal psychology. In particular, it can be viewed as a form of pre-commitment to penalize (by withholding something I would otherwise be tempted to supply) certain behaviors which not only cause people like me to be overlooked and outcompeted for attractive jobs in charities, but also make the charitable world function less efficiently than it other would through a dynamic similar to a tragedy of the commons.

And this tendency I detect in myself really does feel like a precommitment in the sense that (as is true of almost all human precommitments that operate through the emotions) I have no recollection or impression of having chosen it and in the sense that it would probably require the expenditure of a very great deal of mental resources on my part to act contrary to it.

I do not know what you mean by "use a wrench correctly". I mean, I do not see how a person can use a wrench incorrectly.

Can you hint at what I am missing?

ADDED. There is a widespread standard for which way to turn to loosen a bolt and which way to turn to tighten it. Maybe that is what you refer to.

ADDED. If you use the wrong size wrench (i.e., slightly too big) you can "strip" the bolt (i.e., turn the hexagon into a circle, thereby making it impossible to wrench the bolt in the future even with a correctly-sized wrench). And as Romeo mentioned, not everyone is born knowing that the longer the lever, the easier it is for a person to apply a given amount of torque. And you can apply too much torque, thereby stripping the threads, which can lead to failure of the bolt to perform its primary function (holding 2 pieces together) and can make it impossible even to replace the bolt with a fresh bolt. On the other hand, if you do not apply enough torque, the bolt can shake loose. And speaking of shaking loose, you can forget to put in a lock washer where it is needed (with the result that the bolt shakes loose). So yeah, there are things to learn.

Gee, after I put myself in the right frame of mind, I can think of plenty of ways to use a wrench incorrectly. I withdraw the question. I would add however that people should not wait to receive training for using a wrench because it one of those things that can mostly be learned by experience (the purpose of lock washers being an exception to the general rule).

EDITED my first sentence to make it less likely to be interpreted as a potential put-down.

What you say might be true if the only way to do good was to get money from donors. But of course that is not true: a do-gooder can become a donor himself or if he is too poor to donate, he can devote his energies to becoming richer so that he can donate time or money in the future (which is in fact the course that most of the young people inspired by SI's mission are taking).

I am more comfortable speaking about individual altruists rather than charitable organizations. If an individual altruist can find a charity to employ him or find a donor to support his charitable work, then great! If not, then since money is an important resource, he should probably figure out how to get a supply of it. My point in this thread is that if the individual altruist is contemplating spending more than, oh, say 10% or 20% of his life force in becoming more impressive so that he can get a good job at a charity or can get more money from donors, then his plan is probably faulty and that he should instead plan to exchange goods and services he creates for money until money is no longer the constraining resource for his charitable goals.

(For individual altruists who live in countries where it is not as easy to exchange goods and services for money as it is in the English-speaking countries and who cannot emigrate to an English-speaking country, my figure of 10% to 20% might have to be adjusted upward.)

Individuals who make up SI are IMO already investing enough of their time and energy on impressing potential charitable employers, donors and endorsers, hence my request to Holden to clarify what he means when he says, "I feel that [SI] ought to be able to get more impressive endorsements than it has," and, "SI seems to have passed up opportunities to test itself and its own rationality by e.g. aiming for objectively impressive accomplishments."

Many more people would choose to have a paid position with SI than can be given a paid position with SI. What these people who wanted jobs at SI but did not get them usually do is earn as much money as possible with the goal of donating it to the cause. Many of these people are almost as qualified as the people who got jobs at SI. (Although they do not pay much, these are attractive jobs, e.g., because of the quality of the people one gets to spend one's workday with.) It would tend to have a demoralizing effect on those that did not get jobs at SI for the people who did get jobs at SI to spend a significant fraction of their resources consolidating their access to high-status contacts, endorsements, charitable jobs and donor money.

So, not all effort at impressing others is bad, but there is need for a balance.

Although I would have thought that Holden is smart enough to decide whether the FAI project is theoretically sound without his relying on AI experts, maybe I am underestimating the difficulties of people like Holden who are smarter than I am, but who didn't devote their college years to mastering computer science like I did.