I mostly agree with this.

I run into a fair number of epistemologists who are not keen on describing beliefs in terms of probabilities and want to use binary "believe" vs "not believe" terms, or binary "justification." Bayesian updating and utility-maximization decision theory are pretty dominant among philosophers of probability and decision theorists, but not universal among philosophers.

Your scenario is burdened by excessive detail about FAI. Any situation in which people create lots of sims but don't allow lots of suffering/horror in the sims (perhaps as "rescue sims," perhaps because of something like animal welfare laws, or many other possibilities) poses almost the same questions.

Of course people spend their peak years working in those fields. If Eliezer took his decision theory stuff to academia he could pursue that in philosophy. Nick Bostrom's anthropic reasoning work is well-accepted in philosophy. But the overlap is limited. Robin Hanson's economics of machine intelligence papers are not taken seriously (as career-advancing work) by economists. Nick Bostrom's stuff on superintelligence and the future of human evolution is not career-optimal by a large margin on a standard philosophy track.

There's a growing (but still pretty marginal, in scale and status) "machine ethics" field, but analysis related to existential risk or superintelligence is much less career-optimal there than issues related to Predator drones and similar.

Some topics are important from an existential risk perspective and well-rewarded (which tends to result in a lot of talent working on them, with diminishing marginal returns) in academia. Others are important, but less rewarded, and there one needs slack to pursue them (donation funding for the FHI with a mission encompassing the work, tenure, etc).

There are various ways to respond to this. I see a lot of value in trying to seed certain areas, illuminating the problems in a respectable fashion so that smart academics (e.g. David Chalmers) use some of their slack on under-addressed problems, and hopefully eventually make those areas well-rewarded.

Just as an aside, note that Nick Bostrom is in academia in the Future of Humanity Institute at Oxford that he personally founded (as Eliezer founded SIAI) and that has been mostly funded by donations (like the SIAI), mainly those of James Martin. That funding stream allows the FHI to focus on the important topics that they do focus on, rather than devoting all their energy to slanting work in favor of the latest grant fad. FHI's ability to expand with new hires, and even to sustain operations, depends on private donations, although grants have also played important roles. Robin spent many years getting tenure, mostly focused on relatively standard topics.

One still needs financial resources to get things done in academia (and devoting one's peak years to tenure-optimized research in order to exploit post-tenure freedom has a sizable implicit cost, not to mention the opportunity costs of academic teaching loads). The main advantages, which are indeed very substantial, are increased status and access to funding from grant agencies.

Roko wanted it pulled, and it was done by the moderator. Eliezer discussed the expected (in the sense of expectation value, not probable outcome) harm above with many capital letters.

I think everyone at SIAI is unhappy with Roko's post.

Saving someone from being eaten by bears might lead them to conceive the next Hitler, but it probably won't (saith my subjective prior). Even with an infinite future, I assign a substantial probability to hypotheses like:

1. Avoiding human extinction will result in a civilization with an expected positive impact.
2. Particular sorts of human global governance will enable coordination problems to be solved on very large scales.

And so forth. I won't be very confident about the relevant causal connections, but I have betting odds to offer on lots of possibilities, and those let me figure out general directions to go.

Expected values and priors.

From Andy Egan.

The Psychopath Button: Paul is debating whether to press the ‘kill all psychopaths’ button. It would, he thinks, be much better to live in a world with no psychopaths. Unfortunately, Paul is quite confident that only a psychopath would press such a button. Paul very strongly prefers living in a world with psychopaths to dying. Should Paul press the button? (Set aside your theoretical commitments and put yourself in Paul’s situation. Would you press the button? Would you take yourself to be irrational for not doing so?)

Newcomb’s Firebomb: There are two boxes before you. Box A definitely contains $1,000,000. Box B definitely contains $1,000. You have two choices: take only box A (call this one-boxing), or take both boxes (call this two-boxing). You will signal your choice by pressing one of two buttons. There is, as usual, an uncannily reliable predictor on the scene. If the predictor has predicted that you will two-box, he has planted an incendiary bomb in box A, wired to the two-box button, so that pressing the two-box button will cause the bomb to detonate, burning up the $1,000,000. If the predictor has predicted that you will one-box, no bomb has been planted – nothing untoward will happen, whichever button you press. The predictor, again, is uncannily accurate.

I would suggest looking at your implicit choice of counterfactuals and their role in your decision theory. Standard causal decision theory involves local violations of the laws of physics (you assign probabilities to the world being such that you'll one-box, or such that you'll one-box, and then ask what miracle magically altering your decision, without any connection to your psychological dispositions, etc, would deliver the highest utility). Standard causal decision theory is a normative principle for action, that says to do the action that would deliver the most utility if a certain kind of miracle happened. But you can get different versions of causal decision theory by substituting different sorts of miracles, e.g. you can say: "if I one-box, then I have a psychology that one-boxes, and likewise for two-boxing" so you select the action such that a miracle giving you the disposition to do so earlier on would have been better. Yet another sort of counterfactual that can be hooked up to the causal decision theory framework would go "there's some mathematical fact about what decision(decisions given Everett) my brain structure leads to in standard physics, and the predictor has access to this mathematical info, so I'll select the action that would be best brought about by a miracle changing that mathematical fact".