Okay, I think I get it. I was initially thinking that the probabilities of the relationship between MAD and reducing risk being negative, nothing, weak, strong, whatever, would all be similar. If you assume that the probability that we all die without MAD is 50%, and each coin represents a possible probability of death with MAD, then I would have put in one 1% coin, one 2% coin, and so on up to 100. That would give us a distribution just like gwern's given graph.
You're saying that it is very likely that there is no relationship at all, and while surviving provides evidence of a positive relationship over a negative one (if we ignore anthropic stuff, and we probably shouldn't), it doesn't change the probability that there is no relationship. So you'd have significantly more 50% coins than 64% coins or 37% coins to draw from. The updates would look the same, but with only one data point, your best guess is that there is no relationship. Is that what you're saying?
So then the difference is all about prior probabilities, yes? If you have two variables that coorelated one time, and that's all the experimenting that you get to do, how likely is it that they have a positive relationship, and how likely is it that it was a coincidence? I... don't know. I'd have to think about it more.
The standard view of Mutually Assured Distruction (MAD) is something like:
Occasionally people will reply with an argument like:
This is an anthropic argument, an attempt to handle the bias that comes from a link between outcomes and the number of people who can observe them. Imagine we were trying to figure out whether flipping "heads" was more likely than flipping "tails", but there was a coin demon that killed everyone if "tails" came up. Either we would see "heads" flipped, or we would see nothing at all. We're not able to sample from the "tails: everyone-dies" worlds. Even if the demon responds to tails by killing everyone only 40% of the time, we're still going to over-sample the happy-heads outcome.
Applying the anthropic principle here, however, requires that a failure of MAD really would have killed everyone. While it would have killed billions, and made major parts of the world uninhabitable, still many people would have survived. [1] How much would we have rebuilt? What would be the population now? If the cold war had gone hot and the US and USSR had fallen into wiping each other out, what would 2013 be like? Roughly, we're oversampling the no-nukes outcome by the ratio of our current population to the population there would have been in a yes-nukes outcome, and the less lopsided that ratio is the more evidence that MAD did work after all.
[1] For this wikipedia cites: The global health effects of nuclear war (1982), Long-term worldwide effects of multiple nuclear-weapons detonations (1975). Some looking online also turns up an Accelerating Future blog post. I haven't read them thoroughly, and I don't know much about the research here.
I also posted this on my blog