Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Being Half-Rational About Pascal's Wager is Even Worse

18 Post author: Eliezer_Yudkowsky 18 April 2013 05:20AM

For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge payoff is almost certainly doomed in practice.  This kind of clever reasoning never pays off in real life...

...unless you have also underestimated the allegedly tiny chance of the large impact.

For example.  At one critical junction in history, Leo Szilard, the first physicist to see the possibility of fission chain reactions and hence practical nuclear weapons, was trying to persuade Enrico Fermi to take the issue seriously, in the company of a more prestigious friend, Isidor Rabi:

I said to him:  "Did you talk to Fermi?"  Rabi said, "Yes, I did."  I said, "What did Fermi say?"  Rabi said, "Fermi said 'Nuts!'"  So I said, "Why did he say 'Nuts!'?" and Rabi said, "Well, I don't know, but he is in and we can ask him." So we went over to Fermi's office, and Rabi said to Fermi, "Look, Fermi, I told you what Szilard thought and you said ‘Nuts!' and Szilard wants to know why you said ‘Nuts!'" So Fermi said, "Well… there is the remote possibility that neutrons may be emitted in the fission of uranium and then of course perhaps a chain reaction can be made." Rabi said, "What do you mean by ‘remote possibility'?" and Fermi said, "Well, ten per cent." Rabi said, "Ten per cent is not a remote possibility if it means that we may die of it.  If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it."  (Quoted in 'The Making of the Atomic Bomb' by Richard Rhodes.)

This might look at first like a successful application of "multiplying a low probability by a high impact", but I would reject that this was really going on.  Where the heck did Fermi get that 10% figure for his 'remote possibility', especially considering that fission chain reactions did in fact turn out to be possible?  If some sort of reasoning had told us that a fission chain reaction was improbable, then after it turned out to be reality, good procedure would have us go back and check our reasoning to see what went wrong, and figure out how to adjust our way of thinking so as to not make the same mistake again.  So far as I know, there was no physical reason whatsoever to think a fission chain reaction was only a ten percent probability.  They had not been demonstrated experimentally, to be sure; but they were still the default projection from what was already known.  If you'd been told in the 1930s that fission chain reactions were impossible, you would've been told something that implied new physical facts unknown to current science (and indeed, no such facts existed).  After reading enough historical instances of famous scientists dismissing things as impossible when there was no physical logic to say that it was even improbable, one cynically suspects that some prestigious scientists perhaps came to conceive of themselves as senior people who ought to be skeptical about things, and that Fermi was just reacting emotionally.  The lesson I draw from this historical case is not that it's a good idea to go around multiplying ten percent probabilities by large impacts, but that Fermi should not have pulled out a number as low as ten percent.

Having seen enough conversations involving made-up probabilities to become cynical, I also strongly suspect that if Fermi had foreseen how Rabi would reply, Fermi would've said "One percent".  If Fermi had expected Rabi to say "One percent is not small if..." then Fermi would've said "One in ten thousand" or "Too small to consider" - whatever he thought would get him off the hook.  Perhaps I am being too unkind to Fermi, who was a famously great estimator; Fermi may well have performed some sort of lawful probability estimate on the spot.  But Fermi is also the one who said that nuclear energy was fifty years off in the unlikely event it could be done at all, two years (IIRC) before Fermi himself oversaw the construction of the first nuclear pile.  Where did Fermi get that fifty-year number from?  This sort of thing does make me more likely to believe that Fermi, in playing the role of the solemn doubter, was just Making Things Up; and this is no less a sin when you make up skeptical things.  And if this cynicism is right, then we cannot learn the lesson that it is wise to multiply small probabilities by large impacts because this is what saved Fermi - if Fermi had known the rule, if he had seen it coming, he would have just Made Up an even smaller probability to get himself off the hook.  It would have been so very easy and convenient to say, "One in ten thousand, there's no experimental proof and most ideas like that are wrong!  Think of all the conjunctive probabilities that have to be true before we actually get nuclear weapons and our own efforts actually made a difference in that!" followed shortly by "But it's not practical to be worried about such tiny probabilities!"  Or maybe Fermi would've known better, but even so I have never been a fan of trying to have two mistakes cancel each other out.

I mention all this because it is dangerous to be half a rationalist, and only stop making one of the two mistakes.  If you are going to reject impractical 'clever arguments' that would never work in real life, and henceforth not try to multiply tiny probabilities by huge payoffs, then you had also better reject all the clever arguments that would've led Fermi or Szilard to assign probabilities much smaller than ten percent.  (Listing out a group of conjunctive probabilities leading up to taking an important action, and not listing any disjunctive probabilities, is one widely popular way of driving down the apparent probability of just about anything.)  Or if you would've tried to put fission chain reactions into a reference class of 'amazing new energy sources' and then assigned it a tiny probability, or put Szilard into the reference class of 'people who think the fate of the world depends on them', or pontificated about the lack of any positive experimental evidence proving that a chain reaction was possible, blah blah blah etcetera - then your error here can perhaps be compensated for by the opposite error of then trying to multiply the resulting tiny probability by a large impact.  I don't like making clever mistakes that cancel each other out - I consider that idea to also be clever - but making clever mistakes that don't cancel out is worse.

On the other hand, if you want a general heuristic that could've led Fermi to do better, I would suggest reasoning that previous-historical experimental proof of a chain reaction would not be strongly be expected even in worlds where it was possible, and that to discover a chain reaction to be impossible would imply learning some new fact of physical science which was not already known.  And this is not just 20-20 hindsight; Szilard and Rabi saw the logic in advance of the fact, not just afterward - though not in those exact terms; they just saw the physical logic, and then didn't adjust it downward for 'absurdity' or with more complicated rationalizations.  But then if you are going to take this sort of reasoning at face value, without adjusting it downward, then it's probably not a good idea to panic every time you assign a 0.01% probability to something big - you'll probably run into dozens of things like that, at least, and panicking over them would leave no room to wait until you found something whose face-value probability was large.

I don't believe in multiplying tiny probabilities by huge impacts.  But I also believe that Fermi could have done better than saying ten percent, and that it wasn't just random luck mixed with overconfidence that led Szilard and Rabi to assign higher probabilities than that.  Or to name a modern issue which is still open, Michael Shermer should not have dismissed the possibility of molecular nanotechnology, and Eric Drexler will not have been randomly lucky when it turns out to work: taking current physical models at face value imply that molecular nanotechnology ought to work, and if it doesn't work we've learned some new fact unknown to present physics, etcetera.  Taking the physical logic at face value is fine, and there's no need to adjust it downward for any particular reason; if you say that Eric Drexler should 'adjust' this probability downward for whatever reason, then I think you're giving him rules that predictably give him the wrong answer.  Sometimes surface appearances are misleading, but most of the time they're not.

A key test I apply to any supposed rule of reasoning about high-impact scenarios is, "Does this rule screw over the planet if Reality actually hands us a high-impact scenario?" and if the answer is yes, I discard it and move on.  The point of rationality is to figure out which world we actually live in and adapt accordingly, not to rule out certain sorts of worlds in advance.

There's a doubly-clever form of the argument wherein everyone in a plausibly high-impact position modestly attributes only a tiny potential possibility that their face-value view of the world is sane, and then they multiply this tiny probability by the large impact, and so they act anyway and on average worlds in trouble are saved.  I don't think this works in real life - I don't think I would have wanted Leo Szilard to think like that.  I think that if your brain really actually thinks that fission chain reactions have only a tiny probability of being important, you will go off and try to invent better refrigerators or something else that might make you money.  And if your brain does not really feel that fission chain reactions have a tiny probability, then your beliefs and aliefs are out of sync and that is not something I want to see in people trying to handle the delicate issue of nuclear weapons.  But in any case, I deny the original premise:  I do not think the world's niches for heroism must be populated by heroes who are incapable in principle of reasonably distinguishing themselves from a population of crackpots, all of whom have no choice but to continue on the tiny off-chance that they are not crackpots.

I haven't written enough about what I've begun thinking of as 'heroic epistemology' - why, how can you possibly be so overconfident as to dare even try to have a huge positive impact when most people in that reference class blah blah blah - but on reflection, it seems to me that an awful lot of my answer boils down to not trying to be clever about it.  I don't multiply tiny probabilities by huge impacts.  I also don't get tiny probabilities by putting myself into inescapable reference classes, for this is the sort of reasoning that would screw over planets that actually were in trouble if everyone thought like that.  In the course of any workday, on the now very rare occasions I find myself thinking about such meta-level junk instead of the math at hand, I remind myself that it is a wasted motion - where a 'wasted motion' is any thought which will, in retrospect if the problem is in fact solved, not have contributed to having solved the problem.  If someday Friendly AI is built, will it have been terribly important that someone have spent a month fretting about what reference class they're in?  No.  Will it, in retrospect, have been an important step along the pathway to understanding stable self-modification, if we spend time trying to solve the Lobian obstacle?  Possibly.  So one of these cognitive avenues is predictably a wasted motion in retrospect, and one of them is not.  The same would hold if I spent a lot of time trying to convince myself that I was allowed to believe that I could affect anything large, or any other form of angsting about meta.  It is predictable that in retrospect I will think this was a waste of time compared to working on a trust criterion between a probability distribution and an improved probability distribution.  (Apologies, this is a technical thingy I'm currently working on which has no good English description.)

But if you must apply clever adjustments to things, then for Belldandy's sake don't be one-sidedly clever and have all your cleverness be on the side of arguments for inaction.  I think you're better off without all the complicated fretting - but you're definitely not better off eliminating only half of it.

And finally, I once again state that I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.  We live on a planet with upcoming prospects of, among other things, human intelligence enhancement, molecular nanotechnology, sufficiently advanced biotechnology, brain-computer interfaces, and of course Artificial Intelligence in several guises.  If something has only a tiny chance of impacting the fate of the world, there should be something with a larger probability of an equally huge impact to worry about instead.  You cannot justifiably trade off tiny probabilities of x-risk improvement against efforts that do not effectuate a happy intergalactic civilization, but there is nonetheless no need to go on tracking tiny probabilities when you'd expect there to be medium-sized probabilities of x-risk reduction.  Nonetheless I try to avoid coming up with clever reasons to do stupid things, and one example of a stupid thing would be not working on Friendly AI when it's in blatant need of work.  Elaborate complicated reasoning which says we should let the Friendly AI issue just stay on fire and burn merrily away, well, any complicated reasoning which returns an output this silly is automatically suspect.

If, however, you are unlucky enough to have been cleverly argued into obeying rules that make it a priori unreachable-in-practice for anyone to end up in an epistemic state where they try to do something about a planet which appears to be on fire - so that there are no more plausible x-risk reduction efforts to fall back on, because you're adjusting all the high-impact probabilities downward from what the surface state of the world suggests...

Well, that would only be a good idea if Reality were not allowed to hand you a planet that was in fact on fire.  Or if, given a planet on fire, Reality was prohibited from handing you a chance to put it out.  There is no reason to think that Reality must a priori obey such a constraint.

EDIT:  To clarify, "Don't multiply tiny probabilities by large impacts" is something that I apply to large-scale projects and lines of historical probability.  On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody's dumping effort into it then you should dump more effort than currently into it.  On a smaller scale, to compare two x-risk mitigation projects in demand of money, you need to estimate something about marginal impacts of the next added effort (where the common currency of utilons should probably not be lives saved, but "probability of an ok outcome", i.e., the probability of ending up with a happy intergalactic civilization).  In this case the average marginal added dollar can only account for a very tiny slice of probability, but this is not Pascal's Wager.  Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginally increased probabilities of success per added small unit of effort.  It would only be Pascal's Wager if the whole route-to-an-OK-outcome were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.

Comments (168)

Comment author: [deleted] 18 April 2013 06:46:48PM *  27 points [-]

In summary, you could say that I'm in this field less because of what you could do with a quantum computer, than because of what the possibility of quantum computers already does to our conception of the world. Either practical quantum computers can be built, and the limits of the knowable are not what we thought they are; or they can't be built, and the principles of quantum mechanics themselves need revision; or there's a yet-undreamt method to simulate quantum mechanics efficiently using a conventional computer. All three of these possibilities sound like crackpot speculations, but at least one of them is right!

  • Scott Aaronson, in the preface of "Quantum Computing Since Democritus"

Ideally, you put yourself in a scenario where verifying any possibility has a huge payoff.

Comment author: [deleted] 18 April 2013 06:37:29PM 7 points [-]

But Fermi is also the one who said that nuclear energy was fifty years off in the unlikely event it could be done at all, two years (IIRC) before Fermi himself oversaw the construction of the first nuclear pile.

For something in the same(-ish) reference class where the pessimists turned out to be right, commercially viable power generation from nuclear fusion has been “30 years in the future” ever since the mid-20th century.

Comment author: ciphergoth 18 April 2013 08:21:04PM 6 points [-]

Everyone tells this story; I'd like to see a cite. Fusion advocates tell a different story: that fusion was always some large number of dollars away, but the dollars weren't there until relatively recently. Once the dollars arrived, a roadmap was set out and has AFAICT basically hit all its deadlines, with JET, ITER and next DEMO proceeding as planned.

Comment author: CarlShulman 18 April 2013 09:16:57PM *  6 points [-]

Fusion advocates tell a different story: that fusion was always some large number of dollars away, but the dollars weren't there until relatively recently.

Could you link to them?

Comment author: ciphergoth 19 April 2013 12:04:57PM 13 points [-]

I didn't keep links when I read these things, so this is the result of a quick Google search for 'fusion "years away" "dollars away"':

The actual reason is mainly funding. People always use the "twenty/thirty/fifty years away" comment as an insult, a way of showing how fusion (or science in general) is unreliable. The reality is that when those predictions were first made in the 1970s in the wake of the Oil Crisis. What happened during the Oil Crisis? We freaked out (rightly so) and planned to allocate a huge amount of money towards fusion research. What happened after the Oil Crisis ended? That money disappeared. Essentially, scientists were promised X billions of dollars to make fusion work, and said they could do it in a couple decades. Then that money was taken away, and people expected them to stay on schedule. Of course, fusion power turned out to be a lot more complicated than we expected. But the real reason is we simply aren't paying for it. Its not "30 years away" its more like $80 billion dollars away. http://imgur.com/sjH5r

http://www.reddit.com/r/science/comments/1bqxaq/scientists_develop_fusion_rocket_technology_in_lab/c99nvl8

Comment author: Cyan 20 April 2013 02:58:29AM 2 points [-]

I learned something. Excellent.

Comment author: JoshuaZ 29 April 2013 01:18:53PM 0 points [-]

Note that these predictions didn't start in the 1970s. In the 1950s and 1960s similar predictions were made with "20 years away" which is an even shorter timespan. See for example here.

Comment author: gwern 29 April 2013 06:41:15PM 4 points [-]

Your link is to Life, as mainstream a publication as you could find in 1950s. We are all here first-hand familiar with how reporters simplify, misunderstand, and misreport technical matters.

As for your specific quote: I assume you're referring to pg180, the Bhabha quote? The reporter says specifically "a controlled thermonuclear reactor" was <20 years away. He didn't say economical power, power too cheap to meter, break-even or net power, or anything. Was this version of what Bhabha said actually wrong? By 1976, was there nowhere in the world a research tokamak or something which created thermonuclear reactions under controlled non-bomb conditions? I suspect there was.

Comment author: JoshuaZ 29 April 2013 07:09:00PM *  1 point [-]

Those are very good points. The first point isn't that convincing by itself since there are other similar example statements from the 1950s and 60s (although I don't have them available off-hand), and while we do frequenly criticize reporters for misreporting on science matters, most of their statements are not very far off from what is being described. Misreporting is while egregious, a small fraction of most science reporting.

Your second point seems more persuasive. By 1976, not only were there functioning tokamaks, but there were other fusion devices also such as fusors. So the prediction of controlled thermonuclear reactors in 20 years did come true, not just for tokamahs but for other fusion methods as well. This substantially reduces the validity of my point.

Comment author: gwern 29 April 2013 07:50:02PM *  2 points [-]

and while we do frequenly criticize reporters for misreporting on science matters, mos of their statements are not very far off from what is being described

I dunno, sometimes they are completely wrong. A few days ago I got the writer of http://arstechnica.com/tech-policy/2012/06/fbi-halted-one-child-porn-inquiry-because-tor-got-in-the-way/ to massively edit the middle of the article because the original source document explicitly said the child porn was not on Silk Road... and his article said the child porn was on Silk Road. Which is about as wrong as possible. And this is far from the first example of the media getting technological or scientific things completely wrong, which is why you need to read the comments or read the original papers if you're going to base any beliefs on what you're seeing.

It's not hard to make the reported versions of stories or predictions be completely wrong, especially in the context of fusion where we were originally discussing the claims of fusion reporters that the credible published official estimate from the government report of 20-30 years were indeed real but had been made explicitly on the basis of enormous funding increases which never materialized, funding was cut substantially, and actual progress has been better than predicted by the low-funding scenarios. (I put a request in the research help page for a copy of the original report to see if the presented graph is accurate but it hasn't come yet.) It's very easy to slide from the apparently accurate version of the conditional prediction "We predict economical fusion in 30 years if we get the planned funding of $80 billion" to the version "they predict fusion in 30 years".

Comment author: Dr_Manhattan 21 April 2013 05:48:54PM 4 points [-]

SImilar story is told about fission weaponry.

1939, Nobel Prize-winning physicist Niels Bohr had argued that building an atomic bomb "can never be done unless you turn the United States into one huge factory." Years later, he told his colleague Edward Teller, "I told you it couldn't be done without turning the whole country into a factory. You have done just that."

http://energy.gov/em/about-us/em-history

Comment author: wedrifid 18 April 2013 08:26:16AM 6 points [-]

It is predictable that in retrospect I will think this was a waste of time compared to working on a trust criterion between a probability distribution and an improved probability distribution. (Apologies, this is a technical thingy I'm currently working on which has no good English description.)

Cool. Are you or one of your minions likely to write it up in an informal technical way at some point in the not-excessively-distant future?

Comment author: Eliezer_Yudkowsky 18 April 2013 05:14:51PM 8 points [-]

It's looking more likely to be formal actually.

Comment author: wedrifid 18 April 2013 05:39:01PM 4 points [-]

It's looking more likely to be formal actually.

Great, even better.

Comment author: komponisto 18 April 2013 06:19:47AM 5 points [-]

Since I just posted to announce a meetup featuring Michael Vassar, I suppose I was primed to recall his take on the Fermi episode:

...1 in 10 is not such a bad estimate. The problem was not that Fermi was stupid or that he was bad at making estimates; he was probably much better at making estimates than almost everyone. The problem is that he was adhering to a set of rules for what you should be thinking about or talking about that is flat-out insane, frankly. A set of rules that says you shouldn't think about anything until you're ready to do experiments with more-or-less established experimental techniques.

From this perspective -- which assumes that Fermi arrived at his estimate through an honest, non-motivated calculation -- what Fermi should have done was believe his own estimate, instead of applying the heuristic of "if it's not established. experimentally-tested science, it doesn't exist". Because a 10% probability of the scenario in question is indeed approximately 100%: that is, enough to take seriously.

Comment author: Eliezer_Yudkowsky 18 April 2013 06:23:56AM 2 points [-]

Why assign a 90% probability to chain reactions being impossible or unfeasible? How should Fermi have known that, especially when it was false?

EDIT: Be careful with your arguments that Fermi should have assigned the false fact 'chain reactions are impossible' an even more extreme probability than 90%. You are training your brain to assign higher and more extreme probabilities to things that are false. You should be looking for potential heuristics that should have fired in the opposite direction. There's such a thing as overfitting, but there's also such a thing as being cleverly contrarian about reasons why nobody could possibly have figured out X and thus training your brain in the opposite direction of each example.

Comment author: orthonormal 18 April 2013 02:18:21PM 13 points [-]

Because ordinary matter is stable, and the Earth (and, for more anthropically stable evidence, the other planets) hadn't gone up in a nuclear chain reaction already?

Without using hindsight, one might presume that a universe in which nuclear chain reactions were possible would be one in which it happened to ordinary matter under normal conditions, or else only to totally unstable elements, not one in which it barely worked in highly concentrated forms of particular not-very-radioactive isotopes. This also explains his presumption that even if it worked, it would be highly impractical: given the orders of magnitude of uncertainty, it seemed like "chain reactions don't naturally occur but they're possible to engineer on practical scales" is represented by a narrow band of the possible parameters.

I admit that I don't know what evidence Fermi did and didn't have at the time, but I'd be surprised if Szilard's conclusions were as straightforward an implication of current knowledge as nanotech seems to be of today's current knowledge.

Comment author: roystgnr 18 April 2013 04:43:37PM 14 points [-]

Strictly speaking, chain reactions do naturally occur, they're just so rare that we never found one until decades after we knew exactly what we were looking for, so Fermi certainly didn't have that evidence available.

Also, although I like your argument... wouldn't it apply as well to fire as it does to fission? In fact we do have a world filled with material that doesn't burn, material that oxidizes so rapidly that we never see the unoxidized chemical in nature, and material that burns only when concentrated enough to make an ignition self-sustaining. If forests and grasslands were as rare as uranium, would we have been justified in asserting that wildfires are likely impossible?

One reason why neither your argument nor my analogy turned out to be correct: even if one material is out of a narrow band of possible parameters, there are many other materials that could be in it. If our atmosphere was low-oxygen enough to make wood noncombustable, we might see more plants safely accumulating more volatile tissues instead. If other laws of physics made uranium too stable to use in technology, perhaps in that universe fermium would no longer be too unstable to survive in nature.

Comment author: orthonormal 18 April 2013 09:17:51PM 3 points [-]

Good counter-analogy, and awesome Wikipedia article. Thanks!

Comment author: Eliezer_Yudkowsky 18 April 2013 05:37:21PM 6 points [-]

Consider also the nature of the first heap: Purified uranium and a graphite moderator in such large quantities that the neutron multiplication factor was driven just over one. Elements which were less stable than uranium decayed earlier in Earth's history; elements more stable than this would not be suitable for fission. But the heap produced plutonium by its internal reactions, which could be purified chemically and then fizzed. All this was a difficult condition to obtain, but predictable that human intelligence would seek out such points in possibility-space selectively and create them - that humans would create exotic intermediate conditions not existing in nature, by which the remaining sorts of materials would fizz for the first time, and that such conditions indeed might be expected to exist, because among some of the materials not eliminated by 5 billion years, there would be some unstable enough to decay in 50 billion years, and these would be just-barely-non-fizzing and could be pushed along a little further by human intervention, with a wide space of possibilities for which elements you could try. Or to then simplify this conclusion: "Of course it wouldn't exist in nature! Those bombs went off a long time ago, we'll have to build a slightly different sort! We're not restricted to bombs that grow on trees." By such reasoning, if you had attended to it, you might have correctly agreed with Szilard, and been correctly skeptical of Fermi's hypothetical counterargument.

Not taking into account that engineering intelligence will be applied to overcome the first hypothetical difficulty is, indeed, a source of systematic directional pessimistic bias in long-term technological forecasts. Though in this case it was only a decade. I think if Fermi had said that things were 30 years off and Szilard had said 10, I would've been a tad more sympathetic toward Fermi because of the obvious larger reference class - though I would still be trying not to update my brain in the opposite direction from the training example.

Comment author: private_messaging 19 April 2013 09:43:39PM *  4 points [-]

because among some of the materials not eliminated by 5 billion years, there would be some unstable enough to decay in 50 billion years, and these would be just-barely-non-fizzing and could be pushed along a little further by human intervention

Except there aren't any that are not eliminated by, say, 10 billion years. And even 40 million years eliminate everything you can make a nuke out of except U235 . This is because besides fizzling, unstable nuclei undergo this highly asymmetric spontaneous fission known as alpha decay.

Comment author: private_messaging 18 April 2013 04:34:32PM *  6 points [-]

Because they didn't know if fission produced enough prompt neutrons, which is clear from the quoted passage, and probably also because Fermi has estimated that there's on the order of 10 other propositions about the results from fission which he, if presented with them by an equally enthusiastic proponent, would find comparably plausible. I'm thinking that in the alternate realities where fission does something other than producing sufficient number of neutrons (about 3 on the average), you'd assign likewise high number to them by hindsight, with a sum greater than 1 (so stop calling it probability already).

Comment author: komponisto 18 April 2013 06:33:52AM *  4 points [-]

If you view the 90% number as an upper bound, with a few bits' worth of error bars, it doesn't look like such a strong claim. If Szilard and Fermi both agreed that the probability of the bad scenario was 10% or more, then it may well have been dumb luck that Szilard's estimate was higher. Most of the epistemic work would have been in promoting the hypothesis to the 10% "attention level" in the first place.

(Of course, maybe Fermi didn't actually do that work himself, in which case it might be argued that this doesn't really apply; but even if he was anchoring on the fact that others brought it to his attention, that was still the right move.)

Comment author: Eliezer_Yudkowsky 18 April 2013 05:14:09PM -2 points [-]

I suppose if we postulate that Szilard and Rabi did better by correlated dumb luck, then we can avoid learning anything from this example, yes.

Comment author: lukeprog 18 April 2013 07:01:04PM 4 points [-]

Quoted in 'The Making of the Atomic Bomb' by Richard Rhodes

Specifically, on page 280 of the 25th Anniversary Edition of the book.

Comment author: Yvain 18 April 2013 06:46:53AM *  14 points [-]

Are you classifying 10% as a Pascal-level probability? How big does a probability have to get before you don't think Pascal-type considerations apply to it?

Are you suggesting that if there was (for example) a ten percent probability of an asteroid hitting the Earth in 2025, we should devote fewer resources to asteroid prediction/deflection than simple expected utility calculations would predict?

Comment author: orthonormal 18 April 2013 02:20:38PM 4 points [-]

No, he's saying that 10% and 1% are non-Pascalian probabilities for x-risks, but that 1-in-10,000 is effectively Pascalian.

Comment author: ciphergoth 18 April 2013 08:35:27PM 4 points [-]

I don't think it counts as "Pascalian" until it starts to scrape below the threshold of probabilities you can meaningfully assert about propositions. If we were basically assured of a bright astronomical future so long as person X doesn't win the lottery, I wouldn't say that worrying that X might win the lottery was a Pascalian risk.

Comment author: Jonathan_Graehl 18 April 2013 07:51:46AM 0 points [-]

I didn't like his anecdote, either.

I think you've read him wrong. He's opposed to "don't pay attention to high utility * small probability scenarios", on the basis of heroism.

Comment author: Eliezer_Yudkowsky 18 April 2013 05:04:10PM 5 points [-]

I'm usually fine with dropping a one-time probability of 0.1% from my calculations. 10% is much too high to drop from a major strategic calculation but even so I'd be uncomfortable building my life around one. If this was a very well-defined number as in the asteroid calculation then it would be more tempting to build a big reference class of risks like that one and work on stopping them collectively. If an asteroid were genuinely en route, large enough to wipe out humanity, possibly stoppable, and nobody was doing anything about this 10% probability, I would still be working on FAI but I would be screaming pretty loudly about the asteroid on the side. If the asteroid is just going to wipe out a country, I'll make sure I'm not in that country and then keep working on x-risk.

Comment author: shminux 18 April 2013 04:14:47PM 3 points [-]

Can someone point to MIRI's estimates (with justifications) of various x-risks and the odds of mitigating them? Just wondering how, in MIRI's view, the FAI work stacks up against other disaster prevention efforts. I can't seem to find this information on their site.

Comment author: Jack 18 April 2013 06:36:37PM 6 points [-]

One relevant point is that successfully building an FAI mitigates other x-risks.

Comment author: DaFranker 19 April 2013 02:12:59PM 0 points [-]

A potentially-relevant sub-point is that successfully building a fast explosive fooming FAI optimally reduces all x-risks.

Comment author: CarlShulman 18 April 2013 08:19:02AM 6 points [-]

In the course of any workday, on the now very rare occasions I find myself thinking about such meta-level junk instead of the math at hand, I remind myself that it is a wasted motion - where a 'wasted motion' is any thought which will, in retrospect if the problem is in fact solved, have not have contributed to having solved the problem.

If you rule out doing anything except X, then you won't get much out of accurately evaluating the plausibility of X. The point of considering likelihood of success is that there are always other options, including cutting one's losses. But to rule out all competing options requires some assessment of their plausibility relative to X.

Comment author: Eliezer_Yudkowsky 18 April 2013 05:16:53PM 8 points [-]

A lot of meta-level fretting has the property of being one-sided - it's about a single option considered in isolation, not about two alternatives. If there's a concrete alternative that's supposed to help humanity more and has a decent chance of being actually correct vs. the sort of thing one dutifully ought to consider, I am usually totally happy to consider it. (You've seen me ask 'Can we have a concrete policy implication, please?' or 'Is there an option on the table for what we should be doing instead, if that's true?' at a number of discussions, right? This is often what my 'wasted motion' heuristic looks like when it fires.)

Comment author: atorm 18 April 2013 12:14:37PM 9 points [-]

What probability are you assigning to cryonics working that makes you think it's a good idea? I was under the impression that the standard LW argument for signing up was (tiny probability of success)*(monumental heap of utility if it works)=(a good investment). If that's not your argument, what is?

Comment author: [deleted] 18 April 2013 12:25:27PM *  3 points [-]

Hanlon has claimed 6% before, based on a Fermi estimate that may or may not mean anything.

The difference between this and Pascal's Wager is that there's no tradeoff between small probability scenarios; there's either assured death or almost-assured-death. The argument is roughly (small probability)*(high utility) > (lost utility of cheap investment).

(But of course see gwern's remarks on the cheapness of the investment.)

Comment author: drethelin 18 April 2013 04:17:52PM 7 points [-]

You can rephrase it as a small probability of revival vs a small probability of REALLY needing that money.

Comment author: orthonormal 18 April 2013 02:42:20PM *  8 points [-]

I was under the impression that the standard LW argument for signing up was (tiny probability of success)*(monumental heap of utility if it works)=(a good investment). If that's not your argument, what is?

The standard LW argument is that cryonics has a non-tiny probability of success. I did my own estimate, and roughly speaking, P(success) is at least P(A)P(B|A)P(C|A,B)P(D|A,B,C), where

  • A = materialism, i.e. preserving the brain or just the information of the brain-state is enough to preserve the mind
  • B = the freezing process preserves the information of the brain-state in such a way that plausible future tech can recover it
  • C = future society develops the tech to actually recover the info from cryopreserved brains without extravagant energy costs
  • D = my cryonics provider keeps me frozen for the entire time, and someone in the future sees fit to revive me

And my honest estimates were, roughly, P(A) > .95, P(B|A) > .8, P(C|A,B) > .3, and P(D|A,B,C) > .2, giving an overall lower-bound estimate of about 5% (with a lot of metauncertainty, obviously); then I tried to estimate how much waking up in the future would really be worth to me in terms of my current values compared to the value of money now, and overall determined that it was worth it for me to sign up (but not overwhelmingly; had it been 10 times the actual cost of $20 a month for the insurance and dues, I'd have waited until I was significantly richer).

And the point of the post is that 5% is not a Pascal's Wager-esque tiny probability, but something on the level of health risks that we do in fact take seriously.

(You're welcome to take exception to my estimates, of course, but the main point is that I signed up for non-Pascalian reasons. There's actually more to consider, including the effect of MWI and anthropic reasoning on C and D, and consideration about the motivations that someone might revive cryopreserved people, but this is a good enough sketch for current purposes.)

Comment author: jaibot 18 April 2013 03:51:52PM 8 points [-]

E (should go between A and B given your chronological ordering scheme): You die in such a way that high-quality vitrification/plastination is possible. (This variable gets overlooked way too frequently in these calculations).

Comment author: orthonormal 18 April 2013 04:29:24PM 3 points [-]

Ah, good call. For a young and healthy person like me, that's a significant factor, since the likely causes of untimely death would probably be unexpected and/or violent. (Anyone have an idea about how to estimate this one properly?)

Comment author: DaFranker 18 April 2013 06:00:14PM *  2 points [-]

(Anyone have an idea about how to estimate this one properly?)

Get some base rate cause-of-death statistics for people in your age group and geography. Exclude those deaths for which you are certain you are exempt (or just discount them appropriately according to your beliefs that you might die of them, but that's a lot of work and the uncertainty is already so large that this won't affect much of anything).

The hard work is finding good comprehensive stats on this. The WHO databases could be good fallback if you don't have anything better / more specific. For Canada, these proved quite useful to get a general picture.

I did some research for myself, and came to the conclusion that E is (probably) low enough until some age group that I shouldn't bother with cryonics until then. For my specifics, the rough base rates for sudden or destructive death are above 50%, while it's down to something like 15% at 45-54.

The actual math for deciding that I used ended up having a few more factors, but overall what I've got is "don't sign up for cryonics until 40+ unless some other evidence comes up (or the price goes down)".

Comment author: [deleted] 18 April 2013 06:45:40PM 2 points [-]

I did some research for myself, and came to the conclusion that E is (probably) low enough until some age group that I shouldn't bother with cryonics until then.

On the other hand, while E increases with age, so does the cost of life insurance. On the third hand, so does your income and your net worth.

Comment author: [deleted] 19 April 2013 12:32:57PM 1 point [-]

And on the fourth hand, anti-agathics becoming available while you're still alive would bring E back down.

Comment author: jaibot 18 April 2013 06:39:48PM 2 points [-]

I'm surprised your math came out close enough for a factor of less-than-two to make a difference.

Comment author: DaFranker 18 April 2013 07:24:12PM 2 points [-]

Well, yeah, it wasn't just a factor of less than two. Discounting rates, decrease of marginal u / $, P(B) probably increasing over time, and a few other things came into account.

Not to mention the sheer increase in natural mortality rates - %-of-deaths gives you a ratio by which to cut down odds of success, but deaths-per-population is what counts in calculating expected utility of signing up for cryonics at a given time. These rates climb very sharply past 40, especially for the causes of death that cryonics can actually help with.

Overall though, I must admit (after taking another look at it) that my math is/was full of potential holes to poke at. I may be going over it more carefully at some point in the near future, or I may just end up signing up for cryonics to save myself the trouble and never have to think about it this much again (barring new evidence or other events, of course).

Comment author: Eliezer_Yudkowsky 18 April 2013 05:09:18PM 5 points [-]

Expanding conjunctive probabilities without expanding disjunctive probabilities is another classic form of one-sided rationality. If I wanted to make cryonics look more probable than this, I would individually list out many different things that could go right.

Comment author: [deleted] 20 April 2013 09:15:31AM 3 points [-]

Can you give a few examples?

Comment author: orthonormal 18 April 2013 09:01:45PM 2 points [-]

For the purpose of establishing that it's not a Pascalian probability, it suffices to talk about a lower bound on the main line of reasoning.

Ah, I see that I said "estimate" instead of "lower bound" in the critical place. I'll edit.

Comment author: Luke_A_Somers 18 April 2013 09:28:14PM 1 point [-]

In this case, I'm not seeing the disjunctive possibilities that lead one to sign up for cryo in this particular case. ABCD seem to be phrased pretty broadly, and A and B in particular are already pretty big. Do you mean as an alternate to D that, say, a new cryo provider takes over the abandoned preserved heads before they thaw? Or as an alternate to C, that even though the cost is high, they go ahead and do it anyway?

Beyond that, I only see scenarios that are nice but don't point one toward cryopreservation. Like, time travel scans of dying people meaning no one ever really died is wonderful, but it means getting cryopreserved only did good in that your family wouldn't be QUITE as sad you were gone in the time before they 'died'.

Comment author: Eliezer_Yudkowsky 18 April 2013 10:24:15PM 2 points [-]

Do you mean as an alternate to D that, say, a new cryo provider takes over the abandoned preserved heads before they thaw?

Sure. That happened already once in history (though there was, even earlier, a loss-thaw). It's why all modern cryo organizations are very strict about demanding advance payment, despite their compassionate hearts screaming at them not to let their friends die because of mere money. Sucks to be them, but they've got no choice.

Or as an alternate to C, that even though the cost is high, they go ahead and do it anyway?

Yep. I'd think FAI scenarios would tend to yield that.

Basically I always sigh sadly when somebody's discussing a future possibility and they throw up some random conjunction of conditional probabilities, many steps of which are actually pretty darned high when I look at them, with no corresponding disjunctions listed. This is the sort of thinking that would've led Fermi to cleverly assign a probability way lower than 10% to having an impact, by the time he was done expanding all the clever steps of the form "And then we can actually persuade the military to pay attention to us..." If you're going to be silly about driving down all impact probabilities to something small via this sort of conjunctive cleverness, you'd better also be silly and multiply the resulting small probability by a large payoff, so you won't actually ignore all possible important issues.

Comment author: CarlShulman 18 April 2013 11:17:03PM 1 point [-]

"And then we can actually persuade the military to pay attention to us..."

The government did sit on it for quite a while, delaying the bomb until after the defeat of Germany. Nudges from Britain were important in getting things moving.

Comment author: SilasBarta 19 April 2013 12:18:58PM 1 point [-]

The military "paid attention to them" long before that though.

Comment author: atorm 18 April 2013 03:24:04PM 3 points [-]

Ah. Your reasoning seems to be sound, but I would estimate P(B|A) << 0.8. Thank you for the explanation.

Comment author: orthonormal 18 April 2013 09:13:34PM 2 points [-]

My estimate of P(B|A) would be lower, too, had I not read Drexler's Engines of Creation. The theoretical limits of useful nanotech allow for devices way smaller and more efficient than the behemoths of weakly-bonded amino acid chains that make up our cells, so that cheaply repairing intracellular damage is not unreasonable at that point. (Let alone the alternative of scanning the contents and doing the real work on a computer simulation.)

At that point, it becomes a question not of whether cells are structurally intact, but of whether their inter-relationships at the necessary scale remain stably encoded after the vitrification process. (Since there are several different scales which might be "the necessary scale" for recovering a mind, this does involve some uncertainty.) Pending the outcomes of the BPF Prize and the NEMALOAD Project, I'm pretty optimistic on that front. (See here for my declarations of how I'd update given bad news on either of those projects.)

Comment author: ciphergoth 18 April 2013 08:39:36PM 1 point [-]

Why?

Comment author: atorm 18 April 2013 11:46:10PM 7 points [-]

My understanding of neurobiology (BS in biology, current Plant Biology grad student) leads me to believe that the mind is not stored strictly statically in relationships between neurons, but also in the subcellular states of several proteins. These states are unlikely to be preserved in time for cryopreservation. They probably will be disrupted by the freezing process even if a living brain were to be preserved.

Comment author: ciphergoth 20 April 2013 08:08:06AM 4 points [-]

I need to write my "You appear to be making an argument against the technical feasability of cryonics as a comment on a blog post" blog post. I've already blogged all the pieces, but I need to write the one piece that ties it all together.

Comment author: TitaniumDragon 20 April 2013 11:04:30PM 4 points [-]

There's a lot of good reasons to believe that cyronics is highly infeasible. I agree that P(B|A) is low, and P(D|A,B,C) is also absurdly low. We don't care about starving people in Africa today; what is the likelihood that we care about dead frozen people in the future, especially if we have to spend millions of dollars resurrecting them (as is more than likely the case), especially given how difficult it would be to do so? And that's assuming we can even fix whatever caused the problem in the first place; if they die of brain cancer, how are we supposed to fix them, exactly?

Senility is another issue, as it could potentially permanently destroy portions of your brain, rendering you no longer you.

But really I find the overall probability of everything incredibly bad.

But even if we CAN do it, I suspect that it would not be worth doing because what we'd really be doing is just building a copy of you from a frozen copy most likely, in which case you, personally, are still dead; the fact that a copy of you is running around doesn't really change that, and also raises the problem that they could make any numbers of copies of people, which likely would make them dubious about doing so in the first place.

Comment author: gwern 20 April 2013 11:20:02PM 0 points [-]

what is the likelihood that we care about dead frozen people in the future, especially if we have to spend millions of dollars resurrecting them (as is more than likely the case), especially given how difficult it would be to do so?

This is a standard criticism people come up with after 5 seconds of thought, and a perfect example of http://lesswrong.com/lw/h8n/litany_of_a_bright_dilettante/

Do you really think that no one in cryonics hasn't ever thought - 'wait a second! why would anyone in the future even bother putting in the work?' - and you have successfully exposed a fatal ~70-year-old blindspot in a comment written in a few seconds?

Comment author: ciphergoth 21 April 2013 12:51:36PM 5 points [-]

And it's not necessarily that the replies to this problem are good, but that they are what you need to reply to. There's nothing to be said for making a serve we've already returned; to advance the discussion, you need to actually hit the ball back into our court, by reading and replying to the standard replies to this point.

Comment author: TitaniumDragon 25 April 2013 08:52:16PM 4 points [-]

I have never actually seen any sort of cogent response to this issue. Ever. I see it being brushed aside constantly, along with the magical brain restoration technology necessary for this, but I've never actually seen someone go into why, exactly, anyone would bother to thaw them out and revive them, even if it WAS possible to do. They are, all for all intents and purposes, dead, from a legal, moral, and ethical standpoint. Not only that, but defrosting them has little actual practical benefit - while there is obvious value to the possible cryopreservation of organs, that is only true if there aren't better way of preserving organs for shipment and preservation. As things are today, however, that seems unlikely - we already have means of shipping organs and keeping them alive, and given the current trend towards growing organs, it seems far more likely to me that the actual method will be to grow organs and keep them alive rather than keep them in cryopreservation, and without that technology being worked on, there is pretty much no value at all to developing unfreezing technology.

That means that, realistically speaking, the only purpose of such technology would be, say, shipping humans to another planet, which while probably not really rational from an economic perspective is at least somewhat reasonably likely. But even still that is a different kettle of fish - the technology in question may not resemble present day cryogenics at all, and as such may be utterly useless for unfreezing people from present-day cyrogenic treatments. Once you can prove that people CAN be revived in that way, then there is much more incentive towards cryogenics... but that is not present day cryogenics, and there is no evidence to suggest future cryogenic treatments will be very similar to present ones.

Okay, so even all that technology aside, let's assume, at some point, we do develop this technology for whatever reason. At this point, not only do you have to bear the expense of unfreezing these people, but you also have to bear the expense of fixing whatever is wrong with them (which, I will note, actually killed them in the past), as well as fixing whatever damage was done to them prior to being cryogenically frozen (and lest we forget, 10 minutes without oxygen is very likely to cause irreparable brain damage in humans who survive - let alone humans who are beyond what we in the present day can deal with). This is likely to be very, very expensive indeed, and there is little real incentive for someone in the future to spend their money in this way instead of on something else. You are basically hoping for some rich idiot to not only be capable of doing this, but also being willing to do it and having the legal ability to do so (as, lest we forget, there are laws about playing around with human corpses, and I suspect that it is unlikely they will change positively for frozen people in the future - as if they do change, what are the odds that your frozen body won't be used in some other sort of experiment?).

I have never seen arguments which really address these issues. People wave their hands and talk about nanotechnology and brain uploading, but as someone who has actually dealt with nanotechnology I can tell you that it is not, in fact, magical, nor is it capable of many of the feats people believe it will be capable of, nor will it EVER be capable of many of the feats that people imagine it will be capable of. Nanomachines have to be provided with energy the same as anything else, among other, major issues, and I have some severe doubts about the unfreezing process in the first place due to various issues of thermodynamics and the fact that the bodies are not frozen in a setup which is likely to facilitate unfreezing them.

A lot of cryonics arguments basically boil down to "future technology is magic", and that's a pretty big problem for any sort of rational argumentation. "You can't prove that they won't be able to revive me" can be used for all sorts of terrible arguments, as the burden of proof is on the person making the argument that it IS possible, not on the person holding to the present day "we can't, and see no way to do so."

I mean, you look at things like:

http://www.alcor.org/Library/html/resuscitation.htm

The technology in here is, quite literally, magic. It doesn't exist, and it won't exist. Ever. Things on the level are very dumb; they cannot be intelligent, and they cannot act intelligently, because they are too small, too simple. The bits where they stick stuff into your cells is where things get really ridiculous, but even before then, those little nanomachines are going to have real issues doing what you are hoping for, and would have to be custom built for the task at hand. We're talking enormous expense if it is even possible to do at all, and given the extremely small cryogenic population, the odds of perfecting the technology prior to running out of dead people is not very good. Remember, if the result is brain dead or severely brain damaged, it is still a failure. But even these sorts of nanomachines are very questionable; transistors are only going to get 256 times smaller at most, which makes me question whether said nanomachines can function in the way that is hoped for at all. Of course, this is not necessarily a barrier to, say, a different sort of nanomachine (though they'd be more micromachines really, on the scale of a cell rather than on the scale of large molecules) which was controlled by some sort of external process with the little machines being extensions/remotes of it, but this is still questionable.

Extreme expense, questionable technology (which would have to be custom developed for the purpose), the question of whether cryonics is even a viable technological route for something else for cryogenic revival to piggyback on, likely custom technology for reviving people who have died of things that people no longer die of because of earlier preventative measures (why build something to fix someone with late stage cancer when no one gets late state cancer anymore?), legal problems, the necessity for experimental subjects... all of these things add up to the question of why these hypothetical future people are even going to bother. That's assuming it is even ethical to revive someone who is, say, not genetically engineered and therefore would be at the bottom of the societal heap if they were revived.

Comment author: TitaniumDragon 26 April 2013 07:59:48AM 4 points [-]

Dunning-Kruger and experience with similar religious movements suggests otherwise.

It takes someone who really thinks about most things very little time to come up with very obvious objections to most religious doctrine, and given the overall resemblance of cryonics to religion (belief in future resurrection, belief that donating to the church/cyronics institution will bring tangible rewards to yourself and others in the future, belief in eternal life) its not really invalid to suggest something like that.

Which is more likely - that people are deluding themselves over the possibility of eternal life and don't actually have any real answers to the obvious questions, but conveniently ignore them because they see the upside as being so great, or that this has totally been answered, despite the fact that you didn't even articulate an actual answer to it in your response, or even link to it?

I'm pretty sure that, historically speaking, the former is far more likely than the latter.

If someone comes up to you and starts talking about how you have an immortal soul, if you've spent any time studying medicine or neurobiology at all, or have experience with anyone who has suffered brain damage, it really doesn't take you very long to come up with a good counterargument to people having souls. And people have argued about the nature of being for -thousands- of years, and dubiousness about souls has been around for considerably longer than cryonics has been. And yet, people still believe in souls, despite the fact that a very simple, five minutes of thought counterargument exists and has never been countered.

The fact that you did not have a counter for my argument and instead linked to a page which was meant to be a "take that" directed at me is evidence against you having an actual answer to my query, which is always a bad sign. This is not to say that it doesn't have an answer, but a quick, simple answer (or link) would be no more difficult to find than the litany article.

Indeed, after looking at the Alcor site, and reading around, all I really find are arguments against it. The best argument for it that I've seen is that resurrecting 20th century people might be profitable from an entertainment/educational standpoint, but I find even that to be a weak argument - not only is resuscitating someone for the purpose of entertainment deeply morally repugnant (and likely to be so into the future), but wikipedia and various other sources from the 20th and 21st century are likely to be far more valuable to historians, while writers will benefit more from creating their own characters who are considerably more interesting than real people - and it is considerably cheaper and less morally and legally questionable to do so.

So what is the argument for it? If it is so simple to resolve, then what is the resolution?

As ciphergoth pointed out, there isn't really a good answer here. And that is troubling given that the whole thing is pointless if no one is ever going to bring you back anyway. I was reading one article on Alcor which suggested that, even for a cyronics optimist, the odds of it actually paying off were 15% if he only used his most optimistic numbers - and I think his numbers about the technology are optimistic indeed. That's bad news, especially given the guy is someone who actually thinks that doing cryonics is worthwhile.

Comment author: gwern 26 April 2013 10:34:01PM *  5 points [-]

Dunning-Kruger

You obviously have not actually read the Dunning-Krueger paper and understood what it showed.

and experience with similar religious movements suggests otherwise.

Name three. Like V_V, I suspect that for all that you glibly allude to 'cults' you have no personal experience and you have not acquainted yourself with even a surface summary of the literature, much like you have not bothered to so much as read a cryonics FAQ or book before thinking you have refuted it.

It takes someone who really thinks about most things very little time to come up with very obvious objections to most religious doctrine

And it takes even less time to notice that there are long thorough answers to the obvious objections. Your point here is true, but says far more about you than religion or cryonics; after all, many true things like heliocentrism or evolution have superficial easily thought-of objections which have been addressed in depth. Sometimes they work, sometimes they don't; the argument from evil is probably the single most obvious argument against Western religions, there are countless replies from theists of various levels of sophistication, and while I don't think any of them actually work, I also don't think someone going 'My mother died! God doesn't exist!' is contributing anything whatsoever. What, you think the theists somehow failed to notice that bad things happen? Of course they did notice, so if you want to argue against the existence of God, read up on their response.

Which is more likely - that people are deluding themselves over the possibility of eternal life and don't actually have any real answers to the obvious questions, but conveniently ignore them because they see the upside as being so great, or that this has totally been answered, despite the fact that you didn't even articulate an actual answer to it in your response, or even link to it?

If you had spent less time being arrogant, it might have occurred to you that I see this sort of flip reaction all the time in which people learn of cryonics and in five seconds think they've come up with the perfect objection and refuse to spend any time at all to disconfirm their objection. You are acting exactly like the person who said, "but it's not profitable to revive crypatients! QED you're all idiots and suckers", when literally the first paragraph of the Wikipedia article on ALCOR implies how they attempt to resolve this issue; here's a link to the discussion: http://lesswrong.com/lw/gh5/cryo_and_social_obligations/8d43

Notice how you are acting exactly like cheapviagra. You've come up with an obvious issue for cryonics, and rather than do any - gasp! - research; you commented on it. OK, fine. I then told you that there were replies to the issue, and that you should've known this because the issue is so obvious, and rather than learn a lesson about useful contributions, you are instead self-righteously criticizing me for not willing to drop everything and dig up every answer to every idle passing thought you have!

By the way, a benchmark I've found useful in discussing factual matters or matters with a long pre-existing literature is number of citations and hyperlinks per comment. You're still batting a zero.

I'm pretty sure that, historically speaking, the former is far more likely than the latter.

I'm impressed you've failed to notice that LW is maybe a little different from other sites and we have higher standards, and what happens 'historically' isn't terribly relevant.

The fact that you did not have a counter for my argument and instead linked to a page which was meant to be a "take that" directed at me is evidence against you having an actual answer to my query, which is always a bad sign. This is not to say that it doesn't have an answer, but a quick, simple answer (or link) would be no more difficult to find than the litany article.

Apparently you missed the point. The point was: stop being arrogant. Think for a freaking second about how obvious an argument might be and at least what the reply might be, if you cannot be arsed to look up actual sources. Do us the courtesy of not just thinking your way to your bottom line of 'cryonics sucks' but maybe a step beyond that too.

Indeed, after looking at the Alcor site, and reading around, all I really find are arguments against it. The best argument for it that I've seen is that resurrecting 20th century people might be profitable from an entertainment/educational standpoint

Really? That's the best argument? What did you read, exactly? Where do they say 'there's no reason to revive people except for entertainment'? Or are you just picking out the weakest possible argument because that's what you want to talk about?

As ciphergoth pointed out, there isn't really a good answer here.

What? No! That's not what ciphergoth meant at all! Here is what he said:

And it's not necessarily that the replies to this problem are good, but that they are what you need to reply to.

He did not say the answers were not good. He said, that even if the answers were not good, they are still what you need to deal with. You need to work on your reading comprehension if that's what you got out of his comment. (Shades of Dunning-Kruger, since you brought it up...) Or is this another aspect of you refusing to do any research, bring up the weakest lamest arguments you can find as strawmen, and grossly misinterpret what people have said?

I was reading one article on Alcor which suggested that, even for a cyronics optimist, the odds of it actually paying off were 15% if he only used his most optimistic numbers - and I think his numbers about the technology are optimistic indeed. That's bad news,

No, it's not bad news. It's just news. Expected value is about payoff, cost, and probability. 15% means nothing more and nothing less than 15%; without additional details, it does not mean that something is a good idea and it does not mean something is a bad idea either.

Comment author: ciphergoth 27 April 2013 09:13:58AM 3 points [-]

gwern's interpretation of what I wrote here is entirely correct.

Comment author: pianoforte611 28 April 2013 05:20:38PM 0 points [-]

"what is the likelihood that we care about dead frozen people in the future?"

I wondered that as well when I first heard about cryonics. It is true that society in general won't care about frozen people in the future. But that isn't necessary for cryonics to work. Rather its enough that cryonics organizations care about frozen people.

Why would they care? Because the people running the organization have a vested interested in making their clients live. Among other reasons, the people running the organization might one day be clients as well, so they have to care about the success of the project.

Comment author: ciphergoth 21 April 2013 12:53:03PM 0 points [-]

Actually I think the meat of what I want to say is here: http://blog.ciphergoth.org/blog/2010/02/15/blog-comments-and-article/

Comment author: Tenoke 18 April 2013 04:13:01PM *  0 points [-]

90% confidence intervals the way that I understand the cases(B and C require you to involve your interpretation because they are too interdependent under the default interpretation):

.02< P(B|A) < .2

.6<P(C|A,B)< .8

Comment author: Eliezer_Yudkowsky 18 April 2013 05:07:43PM -1 points [-]

Okay, seriously, how the hell did you get this impression?

Comment author: MarkusRamikin 18 April 2013 07:18:27PM *  3 points [-]

If that's not your argument, what is?

Yeah, did you ever write up a full summary of why you think signing up for cryonics is a good idea? Including, hopefully, not just the information theory stuff,, but also how likely you think it is that you will remain funded and get unfrozen even if the technical problems are all solved, etc. Can't find such an article under the cryonics tag, and I'd love to read such a thing from you.

Comment author: jaibot 18 April 2013 10:47:33PM *  2 points [-]

No numbers, but I think this is the most comprehensive argument Eliezer's made on the subject (it's linked from the LW wiki page on cryonics): http://lesswrong.com/lw/wq/you_only_live_twice/

Comment author: jaibot 18 April 2013 07:20:02PM *  2 points [-]

Not sure why you're getting downvoted, but I was under the impression that you were on the far end of the LW bell curves for cryonics-optimism and AI-pessimism.

Comment author: atorm 18 April 2013 11:48:30PM 6 points [-]

I suspect he's getting downvoted because he didn't answer the question, not even with "I don't think it has a low probability of success" or some other simple response.

Comment author: Luke_A_Somers 18 April 2013 09:31:07PM 0 points [-]

I think it was more like (moderate probability of success)*(monumental heap of utility if it works) = (a great investment) so this argument clearly doesn't apply.

Comment author: James_Miller 18 April 2013 02:13:25PM *  4 points [-]

And finally, I once again state that I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.

So is this a bad reason to give $100 to MIRI:

"MIRI reduces existential risks by a non-tiny probability. My contribution of $100 would increase the chance of MIRI's success, however, by only a tiny probability. Still, multiplying this tiny probability increase by the good that would occur if my $100 did end up making the difference justifies my giving $100 to MIRI."

Comment author: Eliezer_Yudkowsky 18 April 2013 05:29:38PM 6 points [-]

On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody's dumping effort into it then you should dump more effort than currently into it. Calculations of marginal impact in POKO/dollar are sensible for comparing two x-risk mitigation efforts in demand of money, but in this case each marginal added dollar is rightly going to account for a very tiny slice of probability, and this is not Pascal's Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginal probabilities per added unit effort. It would only be Pascal's Wager if the whole route-to-humanity-being-OK were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.

Comment author: James_Miller 18 April 2013 11:12:30PM 11 points [-]

Thanks for answering. I just gave $100 to MIRI.

Comment author: shminux 18 April 2013 04:19:35PM 2 points [-]

Eliezer explains it in his other comment (emphasis mine):

Before scanning, I precommit to renouncing, abjuring, and distancing MIRI from the argument in the video if it argues for no probability higher than 1 in 2000 of FAI saving the world, because I myself do not positively engage in long-term projects on the basis of probabilities that low (though I sometimes avoid doing things for dangers that small). There ought to be at least one x-risk effort with a greater probability of saving the world than this - or if not, you ought to make one. If you know yourself for an NPC and that you cannot start such a project yourself, you ought to throw money at anyone launching a new project whose probability of saving the world is not known to be this small.

Comment author: gothgirl420666 25 April 2013 01:30:15AM 2 points [-]

For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge payoff is almost certainly doomed in practice.

Almost certainly doomed, yes. You might even say doomed 9,999 out of 10,000 times.

Comment author: dgsinclair 18 April 2013 08:43:53PM 2 points [-]

EY: I don't multiply tiny probabilities by huge impacts. I also don't get tiny probabilities by putting myself into inescapable reference classes, for this is the sort of reasoning that would screw over planets that actually were in trouble if everyone thought like that.

But isn't the latter exactly what you are doing with Pascal's wager? Underestimating the existence of God's probability so that you may retreat back to 'tiny probability'?

Comment author: [deleted] 18 April 2013 06:27:01PM 2 points [-]

If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it.

Did “excited” mean something different back then? (If so, I may have misinterpreted a certain line in “All Along the Watchtower” by Bob Dylan.)

Comment author: AlanCrowe 18 April 2013 10:53:37PM 13 points [-]

I retain an eight volume dictionary from 1899 to answer this kind of question. Meaning four is

To arouse the emotions of; agitate or perturb mentally; move: as, he was greatly excited by the news.

One real-life example is

The news of the fall of Calcutta reached Madras, and excited the fiercest and bitterest resentment

Today "exciting" is often contrasted with "boring" and has a positive connotation. (eg "We hoped the football game would be exciting and were disappointed when it was boring.") My old dictionary seems evenly balanced with "excited" being bad and good by turns.

Comment author: DaFranker 19 April 2013 02:11:20PM 1 point [-]

I retain an eight volume dictionary from 1899 to answer this kind of question.

That's some epic skill levels in Arcane Lore right there.

Comment author: gwern 19 April 2013 05:05:13PM *  3 points [-]

Or he could've just downloaded a copy of the OED.

("excited: 1.a Stirred by strong emotion, disturbed, agitated. '1855 Macaulay Hist. Eng. III. 275 "The population of Edinburgh was in an excited state."    1864 Mrs. Carlyle Lett. III. 216 "The excited people‥rushed out to me."    1879 McCarthy Own Times I. 199 "Thiers carried with him much of the excited public feeling of France."'")

Comment author: private_messaging 18 April 2013 08:13:28AM *  5 points [-]

Enrico Fermi said:

Well… there is the remote possibility that neutrons may be emitted in the fission of uranium

and then of course perhaps a chain reaction can be made.

The way I interpret it, he gave a remote possibility to enough neutrons being emitted in the fission of uranium (I guess from the tendency of other things to happen to excess neutrons in the nuclei, such as beta decay), and high probability ("of course") to the chain reaction conditional on the above.

I haven't written enough about what I've begun thinking of as 'heroic epistemology' - why, how can you possibly be so overconfident as to dare even try to have a huge positive impact when most people in that reference class blah blah blah - but on reflection, it seems to me that an awful lot of my answer boils down to not trying to be clever about it.

You know what, before trying a startup project seriously for money, I got clever about precisely this issue, joined TopCoder, and did a programming contest. I kind of think you'd likewise be smart enough to see the utility of reducing the uncertainty, if it was your own money on the line.

But in any case, I deny the original premise: I do not think the world's niches for heroism must be populated by heroes who are incapable in principle of reasonably distinguishing themselves from a population of crackpots, all of whom have no choice but to continue on the tiny off-chance that they are not crackpots.

That's all fine, but the issue is that the donors need to be able to do that, or else, no matter how much you argue that people shouldn't be donating on the basis of small probabilities, low probabilities is all they can have.

Comment author: Jonathan_Graehl 18 April 2013 07:47:08AM *  2 points [-]

Laughing at Fermi for 10% is uncharitable.

It sounds like his heuristic for deciding what avenue of research to follow rejected chain reactions. If, as Eliezer claims, >10% should have been obvious to Fermi if he really thought about it, then we can conclude that he didn't feel a need to think about it, for whatever reason.

I do wish senior/brilliant thinkers wouldn't discourage anyone based on their take of something they haven't really thought about, but that probably doesn't stop the really bold upstarts.

I'd like to understand better why really bright and hard-working people don't bother with real thinking when having discussions about their area of theoretical expertise. I guess a social interaction often demands snap decisions.

"heroic epistemology" - that's what all those now-rich startups used, right?

I've fallen out of love with weighing "if people like me ...". Yes, I'd like to make love, and not war, if I were on a boat with 50 clones of myself. But does this help me decide or achieve anything in reality? Aren't there more interesting things to work on than fantasy? I already am the kind of person I am. (partial rebuttal: Eliezer is just the kind of person who's changed/focused by "if people like me ..." daydreams!)

Comment author: Eliezer_Yudkowsky 18 April 2013 05:32:21PM 3 points [-]

"heroic epistemology" - that's what all those now-rich startups used, right?

It's what AirBNB used. I didn't get a chance to hear about them until they had traction, but I honestly think my general good-idea heuristics would've fired more strongly on this than a lot of conventional wisdom.

Wow, you sure are selective in your charity...

Comment deleted 21 April 2013 06:34:27PM [-]
Comment author: OrphanWilde 18 April 2013 07:42:10AM 0 points [-]

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

I'm having trouble making sense of the quoted section. It makes a lot more sense if that's what they're talking about, especially the "if it means that we may die of it," rather than the possibility of a nuclear reaction in general.

Comment author: CarlShulman 18 April 2013 05:20:02PM 10 points [-]

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

That was Teller. And I think it was more "raised the possibility" than "insisted."

Comment author: Eliezer_Yudkowsky 18 April 2013 05:28:31PM 3 points [-]

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

Not as far as I know. This was considered technically even though it seemed obviously false on its face, assigned an even lower credence afterward, and then it didn't in fact turn out to be true.

Comment author: Dr_Manhattan 19 April 2013 08:17:54PM -2 points [-]

I always wondered if Szilard's slightly outcast status (if my recollection of Rhodes book is correct) helped him see things establishment scientists ignored.