Perhaps this criticism of the California governor assumes an over-naive probabilistic modelling, with only two events ("no acts of espionage" => "fifth column exists [or not]"). In reality, there existed some non-public information about an existing japanese spy network (MAGIC decodes; informants) that is unlikely to have been mentioned in a public hearing.
Perhaps the reasoning was more like this: "We know that they are already here. We know that some fraction of the population sympathizes with the mother nation. If the fifth column did not exist in an organized form, we might have seen some sabotage already. Since there hasn't been any, maybe they are holding back for a major strike."
Frank: It is impossible for A and ~A to both be evidence for B. If a lack of sabotage is evidence for a fifth column, then an actual sabotage event must be evidence against a fifth column. Obviously, had there been an actual instance of sabotage, nobody would have thought that way- they would have used the sabotage as more "evidence" for keeping the Japanese locked up. It's the Salem witch trials, only in a more modern form- if the woman/Japanese has committed crimes, this is obviously evidence for "guilty"; if they are innocent of any wrongdoing, this too is a proof, for criminals like to appear especially virtuous to gain sympathy.
Lack of sabotage is obviously evidence for a fifth column trying to lull the government, given the fifth column exists, since the opposite - sabotage occuring - is very strong evidence against that.
However lack of sabotage is still much stronger evidence towards the fifth column not existing.
The takeaway is that if you are going to argue that X group is dangerous because they will commit Y act, you cannot use a lack of Y as weak evidence that X exists, because then Y would be strong evidence that X does not exist, and Y is what you are afraid X is going to do!
You would be much better off using the fact that no sabotage occurred as weak evidence that the 5th column was preventing sabotage.
If there is other evidence that suggests the 5th column exists and that they are dangerous, that is the evidence that should be used. Making up non-evidence (which is actually counter evidence) is not the way to go about it. There are ways of handling court cases that must remain confidential (though it would certainly make the court look bad, it is the right way to do it).
A and ~A are not each evidence for B, if B is "there is a fifth column active". In some ways, as I said, they already knew B - it was true. There were questions of degree - how organized? how ready? how many? - for which A and ~A each provide some hints at.
Earl Warren tumbled headlong into the standard conspiracy theory attractor with, I might add, no deleterious effect on his career. This man was later the 14th Chief Justice of the US Supreme Court and has probably had more lasting effect on US society than any single figure of the 20th century. Thanks for the post.
But that's not the point. The point is that Earl Warren's reasoning was invalid. It didn't matter what other evidence he had (Warren certainly did not know about the ultra-classified MAGIC decodes). The particular observation of no sabotage was evidence against, and could not legitimately be worked into evidence for.
I suspect a part of the appeal of this saying comes from a mental unease with conflicting evidence. It is easier to think of the absence of evidence as not evidence at all, rather than as evidence against where the evidence in favor just happens to be much stronger. Perhaps it is a specific case of a general distaste for very small distinctions, especially those close to 0?
Ad hominem argumentation is another example of evidence which is usually weak, but is still evidence.
The particular observation of no sabotage was evidence against, and could not legitimately be worked into evidence for.
You are assuming that there are only two types of evidence, sabotage v. no sabotage, but there can be much more differentiation in the actual facts.
Given Frank's claim, there is a reasoning model for which your claim is inaccurate. Whether this is the model Earl Warren had in his head is an entirely different question, but here it is:
We have some weak independent evidence that some fifth column exists giving us a prior probability of >50%. We have good evidence that some japanese americans are disaffected with a prior of 90%+. We believe that a fifth column which is organized will attempt to make a significant coordinated sabotage event, possibly holding off on any/all sabotage until said event. We also believe that the disaffected who are here, if there is no fifth column would engage is small acts of sabotage on their own with a high probability.
Therefore, if there are small acts of sabotage that show no large scale organization, this is weak evidence of a lack of a fifth column. If there is a significant sabotage event, this is strong evidence of a fifth...
I would agree that the lack of sabotage cannot be argued as support for accepting an increase in the probability of the existence of a fifth column. But it may not be sufficient to lower the probability that there is a fifth column, and certainly may not be sufficient to lower a prior of greater than 50% to below 50%, even assuming that one is a Bayesian.
If sabotage increases the probability, lack of sabotage necessarily decreases the probability.
What's special about 50%?
When you hear someone say "X is not evidence ...", remember that the Bayesian concept of evidence is not the only concept attached to that word. I know my understanding of the word evidence changed as I adopted the Bayesian worldview. My recollection of my prior use of the word is a bit hazy, but it was probably influenced a good deal by beliefs about what a court would admit as evidence.(This is a comment on the title of the post, not on Earl Warren's rationalization).
If sabotage increases the probability, lack of sabotage necessarily decreases the probability.
That's true in the averages, but different types of sabotage evidence may have different effects on the probability, some negative, some positive. It's conceivable, though unlikely, for sabotage to on average decrease the probability.
This is all fine and good, but it does not address what "evidence" is. I cannot gather evidence of extra solar planets (either evidence for or against existence) with my naked eyes. So in this experiment, even though I see no "evidence" of extra solar planets by looking up into the sky, I still do not have evidence of absense, because in fact I have no evidence at all.
Evidence, from the aspect of probability theory, is only meaningful when the experiment is able to differential between existence and absence.
Then the real question beco...
If all you have is some generic crime data, then more crime in a region can indicate that the Mafia is strong. On the other hand, Mafias keep their own neighborhoods, and the Mafia sometimes can suppress police activity through corruption, so a very low crime rate can indicate that the Mafia is strong.
Of course, background details would suggest which of these is indicated by the evidence
Hi Eliezer, That's another great post, I very much enjoyed reading even though there are gaps in my understanding. I'm new here so I have lots to learn. I wonder if you could kindly explain what you mean by: "Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge. " Thanks, Lou
Warren seems to be arguing that, given that we see no sabotage, this confirms that a Fifth Column exists.
This article makes a very good point very well. If E would be evidence for a hypothesis H, then ~E has to be evidence for ~H.
Unfortunately, I think that it is unfair to read Warren as violating this principle. (I say "Unfortunately" because it would be nice to have such an evocative real example of this fallacy.)
I think that Warren's reasoning is more like the following: Based on theoretical considerations, there is a very high probability P(H) that there is a fifth column. The theoretical considerations have to do with the nature of the Japanese–American conflict and the opportunities available to the Japanese. Basically, there mere fact that the Japanese have both means and motive is enough to push P(H) up to a high value.
Sure, the lack of observed sabotage (~E) makes P(H|~E) < P(H). So the probability of a fifth column goes down a bit. But P(H) started out so high that H is still the only contingency that we should really worry about. The only important question left is, Given that there is a fifth column, is it competent or incompetent? Does the obse...
Warren stated in the quote that the lack of any subversive activity was the most convincing factor of all the evidence he has that the 5th Column would soon commit subversive activity.
(Emphasis added.)
I just don't see that in the quote. Here is the Warren quote from the OP:
"I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed... I believe we are just being lulled into a false sense of security."
His claim isn't that subversive activity will start soon. The claim is that subversive activity will be "timed just like Pearl Harbor was timed". I read this to mean that he anticipates a centrally-orchestrated, synchronized, large-scale attack, of the sort that could only be pulled off by a disciplined, highly-competent fifth column.
If they had seen small, piece-meal efforts at sabotage, then that would have been evidence against a competent fifth column. That is, P(there is a competent fifth column | there has been piece-meal sabotage) < P(t...
I have to think that there is another question to be considered: What are the odds that Japanese-Americans would commit sabotage we could detect as sabotage? If the odds are very high that detectable sabotage would occur, then the absence of sabotage would be evidence in favor of something preventing sabotage. A conspiracy which collaborates with potential saboteurs and encourages them to wait for the proper time to strike then becomes a reasonable hypothesis, if such a conspiracy would believe that an initial act of temporally focused sabotage would be effective enough to have greater utility than all the acts of sabotage which would otherwise occur before the time of the sabotage spree.
The problem with this scenario, as presented, is that it assumes that "sabotage" is a binary variable. If that were the case, the pool of possibilities would consist of: (1) Fifth Column exists & sabotage occurs, (2) Fifth Column exists & sabotage does not occur, and (3) Fifth Column does not exist & sabotage does not occur (presuming that sabotage, as defined in the scenario, could only be accomplished by Fifth Column). In that case, necessarily, lack of sabotage could only reduce the probability of (1), and therefore could only redu...
If absence of proof is not proof of absence, but absence of evidence is evidence of absence, what makes proof different from evidence?
Example: we currently have no evidence supporting the existence of planets orbiting stars in other galaxies, because our telescopes are not powerful enough to observe them. Should we take this as evidence that no galaxy except ours has planets around its stars?
Another example: before the invention of the microscope, there was no evidence supporting the existence of bacteria because there were no means to observe them. Should've this fact alone been interpreted as evidence of absence of bacteria (even though bacteria did exist before microscopes were invented)?
Hi DevilMaster, welcome to LessWrong!
Generally, the answer to your question is Bayes' Theorem. This theorem is essentially the mathematical formulation of how evidence ought to be weighed when testing ideas. If the wikipedia article doesn't help you much, Eliezer has written an in-depth explanation of what it is and why it works.
The specific answer to your question can be revealed by plugging into this equation, and defining "proof". We say that nothing is ever "proven" to 100% certainty, because if it were (again, according to Bayes' Theorem), no amount of new evidence against it could ever refute it. So "proof" should be interpreted as "really, really likely". You can pick a number like "99.9% certain" if you like. But your best bet is to scrap the notion of absolute "proof" and start thinking in likelihoods.
You'll notice that an integral part of Bayes' Theorem is the idea of how strongly we would expect to see a certain piece of evidence. If the Hypothesis A is true, how likely is it that we'll see Evidence B? And additionally, how likely would it be to see Evidence B regardless of Hypothesis A?
For a piece of evidence t...
There is more discussion of this post here as part of the Rerunning the Sequences series.
Another proof & discussion: http://kim.oyhus.no/AbsenceOfEvidence.html
I'm pretty sure you just used this as an rhetoric tool, but by bayesian theory, isn't it impossible to construct a hypothesis which allocates a probability of zero to an event? But don't you say exactly that in your text?
even if the alternative hypothesis does not allow it at all
I mean allocating a probability of zero to an event implies that it doesn't matter what evidence is presented to you, the probability of that particular event will never become anything else than zero. And as it is impossible to disprove something in the same way it is impossib...
A simple counter example (hopefully shorter and more clear than the other more in depth criticism by michael sullivan) is the scenario where warren had exactly equal priors for organized fifth column, unorganized fifth column, and no fifth column.
p(organized) = .33
p(unorganized) = .33
p(none) = .33
If he was practically certain that an organized fifth column would wait to make a large attack, and a unorganized fifth column would make small attacks then seeing no small attacks his new probabilities would approximately be:
p(organized) = .5
p(none) = .5
So he wo...
The video game Star Ocean: Til The End Of Time has a model of interstellar society that tries to solve Fermi's conundrum. Planets capable of interstellar travel form an accord that treats less advanced civilizations as nature preserves and agree not to contact or help them. This model does have several problems, such as communication wavelengths would still be visible to us (they have some undiscovered form of communication?) and sufficiently advanced societies should have an ethical dilemma with allowing intelligent species to go through dark ages and pro...
I disagree with the article for the following reason: if I have two hypotheses that both explain an "absence of evidence" occurrence equally well, then that occurrence does not give me reason to favor either hypothesis and is not "evidence of absence."
Example: Vibrams are a brand of toe-shoes that recently settled a big suit because they couldn't justify their claims of health benefits. We have two hypotheses (1) Vibrams work, (2) Vibrams don't work. Now, if a well-executed experiment had been done and failed to show an effect, that wou...
Warren's full speech is available at archive.org: "Unfortunately, however, many of our people and some of our authorities and, I am afraid, many of our people in other parts of the country are of the opinion that because we have had no sabotage and no fifth column activities in this State since the beginning of the war, that means that none have been planned for us. But I take the view that that is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage that we are to get, the fifth column ac...
So is there ever a time where you can use absence of evidence alone to disprove a theory, or do you always need other evidence as well? Because is some cases absence of evidence clearly does not disprove a theory, such as when quantum physics was first being discovered, there was not a lot of evidence for it, but can the inverse ever be true will lack of evidence alone proves the theory is false?
Let E stand for the observation of sabotage
Didn't you mean "the observation of no sabotage"?
More acuratly, "absence of evidence you would expect to see if the statement is true" is evidence of absence.
If there's no evidence you'd expect if the statement is true, absence of evidence is not evidence of absence.
For example, if I tell you I've eaten cornflakes for breakfast, no matter whether or not the statement is true, you won't have any evidence in either direction (except for the statement itself) unless you're willing to investigate the matter (like, asking my roommates). In this case, absence of evidence is n...
Hang on, the Japanese example is flawed. There IS an intelligence branch of the Japanese army; this would be well understood by any tactician. Seeing no evidence to their action, and inferring that this is due to their skill, not an irrational assumption.
The odds form of Bayes Theorem is particularly useful here
which can be intuitively understood as Posterior Odds = Likelihood Ratio × Prior Odds.
It shows us exactly how we should update our belief (prior odds -> posterior odds) based on the likelihood ratio, which is essentially "the odd of evidence appearing if the hypothesis is true vs. not true. It can be interpreted intuitively as "Evidence supports whatever makes it more likely".
In the context of this article, since "no sabotage" is more likely if there is no Fif...
From Robyn Dawes’s Rational Choice in an Uncertain World:
In fact, this post-hoc fitting of evidence to hypothesis was involved in a most grievous chapter in United States history: the internment of Japanese-Americans at the beginning of the Second World War. When California governor Earl Warren testified before a congressional hearing in San Francisco on February 21, 1942, a questioner pointed out that there had been no sabotage or any other type of espionage by the Japanese-Americans up to that time. Warren responded, “I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed . . . I believe we are just being lulled into a false sense of security.”
Consider Warren’s argument from a Bayesian perspective. When we see evidence, hypotheses that assigned a higher likelihood to that evidence gain probability, at the expense of hypotheses that assigned a lower likelihood to the evidence. This is a phenomenon of relative likelihoods and relative probabilities. You can assign a high likelihood to the evidence and still lose probability mass to some other hypothesis, if that other hypothesis assigns a likelihood that is even higher.
Warren seems to be arguing that, given that we see no sabotage, this confirms that a Fifth Column exists. You could argue that a Fifth Column might delay its sabotage. But the likelihood is still higher that the absence of a Fifth Column would perform an absence of sabotage.
Let E stand for the observation of sabotage, and ¬E for the observation of no sabotage. The symbol H1 stands for the hypothesis of a Japanese-American Fifth Column, and H2 for the hypothesis that no Fifth Column exists. The conditional probability P(E | H), or “E given H,” is how confidently we’d expect to see the evidence E if we assumed the hypothesis H were true.
Whatever the likelihood that a Fifth Column would do no sabotage, the probability P(¬E | H1), it won’t be as large as the likelihood that there’s no sabotage given that there’s no Fifth Column, the probability P(¬E | H2). So observing a lack of sabotage increases the probability that no Fifth Column exists.
A lack of sabotage doesn’t prove that no Fifth Column exists. Absence of proof is not proof of absence. In logic, (A ⇒ B), read “A implies B,” is not equivalent to (¬A ⇒ ¬B), read “not-A implies not-B .”
But in probability theory, absence of evidence is always evidence of absence. If E is a binary event and P(H | E) > P(H), i.e., seeing E increases the probability of H, then P(H | ¬ E) < P(H), i.e., failure to observe E decreases the probability of H . The probability P(H) is a weighted mix of P(H | E) and P(H | ¬ E), and necessarily lies between the two.1
Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely to produce the signs. The absence of an observation may be strong evidence of absence or very weak evidence of absence, depending on how likely the cause is to produce the observation. The absence of an observation that is only weakly permitted (even if the alternative hypothesis does not allow it at all) is very weak evidence of absence (though it is evidence nonetheless). This is the fallacy of “gaps in the fossil record”—fossils form only rarely; it is futile to trumpet the absence of a weakly permitted observation when many strong positive observations have already been recorded. But if there are no positive observations at all, it is time to worry; hence the Fermi Paradox.
Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge. The strength of a model is not what it can explain, but what it can’t, for only prohibitions constrain anticipation. If you don’t notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.
1 If any of this sounds at all confusing, see my discussion of Bayesian updating toward the end of The Machine in the Ghost, the third volume of Rationality: From AI to Zombies.