Nick Szabo on acting on extremely long odds with claimed high payoffs:
Beware of what I call Pascal's scams: movements or belief systems that ask you to hope for or worry about very improbable outcomes that could have very large positive or negative consequences. (The name comes of course from the infinite-reward Wager proposed by Pascal: these days the large-but-finite versions are far more pernicious). Naive expected value reasoning implies that they are worth the effort: if the odds are 1 in 1,000 that I could win $1 billion, and I am risk and time neutral, then I should expend up to nearly $1 million dollars worth of effort to gain this boon. The problems with these beliefs tend to be at least threefold, all stemming from the general uncertainty, i.e. the poor information or lack of information, from which we abstracted the low probability estimate in the first place: because in the messy real world the low probability estimate is almost always due to low or poor evidence rather than being a lottery with well-defined odds.
Nick clarifies in the comments that he is indeed talking about singularitarians, including his GMU colleague Robin Hanson. This post appears to revisit a comment on an earlier post:
In other words, just because one comes up with quasi-plausible catastrophic scenarios does not put the burden of proof on the skeptics to debunk them or else cough up substantial funds to supposedly combat these alleged threats.
Typical philosophy: tear strawman alternatives to prove a wild guess. (Why strawman: because can be also dependent on anything else entirely like positions of copies)
Also, the 99% confidence is not in the dependence on non yous (and yous BUT nothing else), the 99% confidence is in the wild guess of independence from everything else and dependence to count of yous, to be wrong. Also, consider two computer circuits real nearby, running identical you, separated by thin layer of dielectric. Remove dielectric, 1 copy with thicker wires. Conclusion: it may depend to thickness of wires of a copy or maybe to the speed of the copy.
Hell, the probability of being in a specific copy may just as well be undefined entirely until a copy figures out which copy it is, and then depend solely to how it was figured out.
Let's suppose that the probability of being sampled out of model is sum of 2^-l over all codes that pluck you out of the model, like in Solomonoff induction. May well be dependent on the presence or absence of stone dummies (provided those break some simple method of locating you). Will definitely depend to your position. Go show it broken.
edit : actually, this alternative distribution for observers (and observer-moments) based on Solomonoff-type prior has been proposed here before by Wei_Dai , and has also been mentioned by Marcus Hutter. I'm not at all impressed by Nick Bostrom, that's the point, or philosophy for that matter. The conclusions of philosophers - given relative uselessness of philosophy compared to science - ought to be taken as very low grade evidence.
Thank you for linking to Hutter's talk, what an astounding mind. What a small world it is, I remember being impressed by him when I sat through his courses back at grad school, little knowing how much of my future perspective on map-building would eventually depend on his and his colleagues' school of thought.
That presentation should be mandatory reading. In all Everett branches.