No, I meant "-log10(X)=pX", in the general sense. That is standard use in chemistry and other fields that regularly deal with very small numbers.
Ah, okay, that's cleared up my linguistic confusion; thanks!
And if your model can't distinguish between something that happens one in ten thousand times and something that happens one in a million times, you aren't simulating a probabilistic infinite series, and you can make a deterministic conclusion about your worst-case scenario.
Sort of. These sorts of simulation projects go on in my department, and so I'm familiar with them. For example, a person that used to work down the hall was working on uncertainty quantification for models of nuclear power plants- we have a relatively good idea of how likely any particular part is to break, and how to simulate what happens if those break, and we can get out a number that says "we think there's a 1.3e-9 chance per year per plant of a catastrophic event," for example, but we want to get out the number that says "if our estimate of the rate at which valves fail is off by 5%, our final estimate will be off by X%", and use that to get a distribution on the final catastrophe estimate. (This will give us a better sense of total risk, as well as where we should focus our experiments and improvements.)
So they can say "our model tells us that, with these point inputs, this happens once in a billion times," but they can't yet say "with our model and these input distributions, the chance of this happening more than once in a million times is less than one in a thousand," which must be true for the first statement to be useful as an upper bound of the estimate (rather than an expected value of the estimate).
The goal is to be able to calculate things currently relegated to simulation.
It's not clear to me what you mean by this, since I see calculation as a subset of simulation. If you mean we'd like to analytically integrate over all unknowns, even if we had the integration power (which we can't), we don't have good enough estimates of the uncertainties for that to be all that much more meaningful than our current simulations. With only, say, a thousand samples, it may be very difficult to determine the thickness of the tails of the population distribution, but the thickness of the tails may determine the behavior of the expected value / the chance of catastrophe. These problems show up at much lower quantiles, and it's not clear to me that just ignoring events more rare than a certain quantile will give us useful results.
To use the bridge example again- for the 'expected load' distribution to be faithful above the 99.99th percentile, that would require that we be surprised less than once every thirty years. I don't think there are traffic engineers with that forecasting ability, and I think their forecasting ability starts to break down more quickly than the rare events we're interested in, so we have to tackle these problems somehow.
Well, when I was working on a S5W/S3G (MTS 635, SSBN 732 blue) power plant, our baseline "end of the world" scenario started with "a non-isolateable double-ended shear of a main coolant loop". (half of the power plant falls off). I can't begin to estimate the likelihood of that failure, but I think quantum mechanics can.
If classical mechanics gives you a failure rate that has uncertainty, you can incorporate that uncertainty into your final uncertainty: "We believe it is four nines or better that this type of valve fails in this ma...
(Mathematicians may find this post painfully obvious.)
I read an interesting puzzle on Stephen Landsburg's blog that generated a lot of disagreement. Stephen offered to bet anyone $15,000 that the average results of a computer simulation, run 1 million times, would be close to his solution's prediction of the expected value.
Landsburg's solution is in fact correct. But the problem involves a probabilistic infinite series, a kind used often on less wrong in a context where one is offered some utility every time one flips a coin and it comes up heads, but loses everything if it ever comes up tails. Landsburg didn't justify the claim that a simulation could indicate the true expected outcome of this particular problem. Can we find similar-looking problems for which simulations give the wrong answer? Yes.
Here's Perl code to estimate by simulation the expected value of the series of terms 2^k / k from k = 1 to infinity, with a 50% chance of stopping after each term.
(If anyone knows how to enter a code block on this site, let me know. I used the "pre" tag, but the site stripped out my spaces anyway.)
Running it 5 times, we get the answers
ave sum=7.6035709716983
ave sum=8.47543819631431
ave sum=7.2618950097739
ave sum=8.26159741956747
ave sum=7.75774577340324
So the expected value is somewhere around 8?
No; the expected value is given by the sum of the harmonic series, which diverges, so it's infinite. Later terms in the series are exponentially larger, but exponentially less likely to appear.
Some of you are saying, "Of course the expected value of a divergent series can't be computed by simulation! Give me back my minute!" But many things we might simulate with computers, like the weather, the economy, or existential risk, are full of power law distributions that might not have a convergent expected value. People have observed before that this can cause problems for simulations (see The Black Swan). What I find interesting is that the output of the program above doesn't look like something inside it diverges. It looks almost normal. So you could run your simulation many times and believe that you had a grip on its expected outcome, yet be completely mistaken.
In real-life simulations (that sounds wrong, doesn't it?), there's often some system property that drifts slowly, and some critical value of that system property above which some distribution within the simulation diverges. Moving above that critical value doesn't suddenly change the output of the simulation in a way that gives an obvious warning. But the expected value of keeping that property below that critical value in the real-life system being simulated can be very high (or even infinite), with very little cost.
Is there a way to look at a simulation's outputs, and guess whether a particular property is near some such critical threshold? Better yet, is there a way to guess whether there exists some property in the system nearing some such threshold, even if you don't know what it is?
The October 19, 2012 issue of Science contains an article on just that question: "Anticipating critical transitions", Marten Scheffer et al., p. 344. It reviews 28 papers on systems and simulations, and lists about a dozen mathematical approaches used to estimate nearness to a critical point. These include:
So if you're modeling global warming, running your simulation a dozen times and averaging the results may be misleading. [1] Global temperature has sudden [2] dramatic transitions, and an exceptionally large and sudden one (15C in one million years) neatly spans the Earth's greatest extinction event so far on the Permian-Triassic boundary [3]. It's more important to figure out what the critical parameter is and where its critical point is than to try and estimate how many years it will be before Manhattan is underwater. The "expected rise in water level per year" may not be easily-answerable by simulation [4].
And if you're thinking about betting Stephen Landsburg $15,000 on the outcome of a simulation, make sure his series converges first. [5]
[1] Not that I'm particularly worried about global warming.
[2] Geologically sudden.
[3] Sun et al., "Lethally hot temperatures during the early Triassic greenhouse", Science 338 (Oct. 19 2012) p.366, see p. 368. Having just pointed out that an increase of .000015C/yr counts as a "sudden" global warming event, I feel obligated to also point out that the current increase is about .02C/yr.
[4] It will be answerable by simulation, since rise in water level can't be infinite. But you may need a lot more simulations than you think.
[5] Better yet, don't bet against Stephen Landsburg.