This is a terrible misrepresentation. SI does not argue for donations on these grounds; Eliezer and other SI staff have explicitly rejected such Pascalian reasons, but instead argued that the risks that they wish to avert are quite probable.
could win $1 billion, and I am risk and time neutral
Who has constant marginal utility of money up to $1,000,000,000?
The biggest problem with these schemes is that, the closer to infinitesimal probability, and thus usually to infinitesimal quality or quantity of evidence, one gets, the closer to infinity the possible extreme-consequence schemes one can dream up
Consequences can't be inflated to make up for arbitrarily low probabilities. Consequences are connected: if averting human extinction by proliferation of morally valueless machinery is super valuable because of future generations, then the gains of averting human extinction by asteroids, or engineered diseases, will be on the same scale.
It cost roughly $100 million to launch a big search for asteroids that has now located 90%+ of large (dinosaur-killer size) asteroids, and such big impacts happen every hundred million years or so, accompanied by mass extinctions, particularly of large animals. If working on AI, before AI is clearly near and better understood, had a lower probability of averting x-risk reduction per unit cost than asteroid defense, or adding to the multibillion dollar annual anti-nuclear proliferation or biosecurity budgets, or some other intervention, then it would lose.
"Some nonzero chance" isn't enough, it has to be a "chance per cost better than the alternatives."
Seems like there is more going on than just "Do transhumanists endorse Pascalian bargains?" Because of confidence levels inside and outside an argument, the fact that someone (e.g. SI) makes an argument that a particular risk has a non-negligible probability does not mean that someone examining this claim should assign a non-negligible probability. It's very possible for someone thinking about (e.g.) AI risk to assign low probabilities and thus find themselves in a Pascalian situation even if SI argues that the probability of AI risk is high.
It can be summarized as follows: for basic reasons of economics and computer science, specialized algorithms are generally far superior to general ones.
It would be better to present, as your main reason, "the kinds of general algorithms that humans are likely to develop and implement, even absent impediments caused by AI-existential risk activism, will almost certainly be far inferior to specialized ones". That there exist general-purpose algorithms which subsume the competitive abilities of all existing human-engineered special-purpose algorithms, given sufficient advantages of scale in number of problem domains, is trivial by the existence proof constituted by the human economy.
Put another way: There is some currently-unsubstitutable aspect of the economy which is contained strictly within human cognition and communication. Consider the case where the intellectual difficulties involved in understanding the essence of this unsubstitutable function were overcome, and it were implemented in silico, with an initial level of self-engineering insight already equal to that which was used to create it, and with starting capital and education sufficient to overcome transient ...
I don't think Nick made a good case as to why these movements / belief systems deserve to be call "scams" and more importantly deserve to be ignored (in favor of "spending more time learning about what has actually happened in the real world"). The fact that certain hopes and threats share certain properties (which Nick numbered 1-3 in his post) is unfortunate, but I didn't find any convincing arguments in his post showing why these hopes and threats should therefore be ignored.
(My overall position, which I'll repeat in case anyone is c...
Then we can pardon Bostrom for not taking them into account.
When pondering the possibilities where we live given lack of grand unified theory 'of everything', you can't assume your physical intuitions hold true. In fact you should assume the opposite. The MWI is just an example of how philosophical argument that looks entirely invariable admits defeat from possible physics, in a way in which the mathematics - which philosophy mimics - does not. That means validity of argument requires validity of intuitions, which are reasonably very unlikely to be valid in any grand sense. There's also historical example: a lot of philosophers assuming Euclidean geometry is the only logically possible kind of geometry, without even noticing that they are making such assumption, up until mathematicians came up with alternative.
What? Could you write this more clearly, I have no idea what you're trying to say.
You assume that the probability of being among either group depends to number of yous within that group (rather than something entirely different), to do anthropic reasoning beyond tautological 'we can't observe universes where we can't be alive'. In my opinion this is a case of wild guess over unknowns, totally false precision.
I came up with a clearer example of how something totally different may actually make a lot more sense: Solomonoff induction on codes that model various universes. A model of the universe outputting the data matching your internal self-perception must not only contain yourself, but must include the code that finds yourself within that model, so that output begins with sense data. It is then clear that the code that picks one of yous out of a model full of all sorts of simulations may easily be larger than the code that picks you out of the world where there is just one you, but the number of various others is much smaller.
I'm not arguing that Bostrom is bad for a philosopher. I am outlining how in philosophy you just make all sorts of assumptions that you don't notice are wild guesses, and how the field is basically built on false precision. I.e. you assume that the probability of being within some huge set full of yous and non-yours is independent of number of non-yous, which is just a wild guess. Connect together half a dozen implicit wild guesses with likelihood of correctness of overly generous 1 in 100 each , and we're speaking of probability of correctness in the range of 10^-12 . Philosophy is generally like this.
I believe this falls under Nick Szabo's complaint about false precision.
Also, a paper by Steven Weinberg on usefulness of philosophy.
It seems to me that funding philosophical works in this field may actually be actively harmful, due to establishment of such false precision and prejudices. It's like funding 'embracing bias, imprecision, and making your mind up before checking where the mathematics will lead you'.
If even Newton looks this foolish, what hope can the rest of us have? No, the suggestion 'would this make me look foolish in 100 years?' does us no good in practice.
That's the whole point: very low probability of being right. There's a crucial difference: the methods Newton employed managed to achieve a non-zero (and not negligible) truth finding rate. So he made something that does not look silly. Even with this, most of stuff was quite seriously wrong.
you assume that the probability of being within some huge set full of yous and non-yours is independent of number of non-yous, which is just a wild guess. Connect together half a dozen implicit wild guesses with likelihood of correctness of overly generous 1 in 100 each
Do you think it's "generous" to assign only 99% probability in the negation of "the probability of being within some huge set full of yous and non-yours is independent of number of non-yous" where "you" is interpreted to include all your observations? That seems like insane overconfidence in a view that goes haywire in simple finite discrete cases.
Nick Szabo on acting on extremely long odds with claimed high payoffs:
Beware of what I call Pascal's scams: movements or belief systems that ask you to hope for or worry about very improbable outcomes that could have very large positive or negative consequences. (The name comes of course from the infinite-reward Wager proposed by Pascal: these days the large-but-finite versions are far more pernicious). Naive expected value reasoning implies that they are worth the effort: if the odds are 1 in 1,000 that I could win $1 billion, and I am risk and time neutral, then I should expend up to nearly $1 million dollars worth of effort to gain this boon. The problems with these beliefs tend to be at least threefold, all stemming from the general uncertainty, i.e. the poor information or lack of information, from which we abstracted the low probability estimate in the first place: because in the messy real world the low probability estimate is almost always due to low or poor evidence rather than being a lottery with well-defined odds.
Nick clarifies in the comments that he is indeed talking about singularitarians, including his GMU colleague Robin Hanson. This post appears to revisit a comment on an earlier post:
In other words, just because one comes up with quasi-plausible catastrophic scenarios does not put the burden of proof on the skeptics to debunk them or else cough up substantial funds to supposedly combat these alleged threats.