Tyrrell_McAllister comments on Absence of Evidence Is Evidence of Absence - Less Wrong

54 Post author: Eliezer_Yudkowsky 12 August 2007 08:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (108)

Tags: bayesian

Comments (108)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Tyrrell_McAllister 15 December 2010 07:18:32PM * 9 points [-]

Warren seems to be arguing that, given that we see no sabotage, this confirms that a Fifth Column exists.

This article makes a very good point very well. If E would be evidence for a hypothesis H, then ~E has to be evidence for ~H.

Unfortunately, I think that it is unfair to read Warren as violating this principle. (I say "Unfortunately" because it would be nice to have such an evocative real example of this fallacy.)

I think that Warren's reasoning is more like the following: Based on theoretical considerations, there is a very high probability P(H) that there is a fifth column. The theoretical considerations have to do with the nature of the Japanese–American conflict and the opportunities available to the Japanese. Basically, there mere fact that the Japanese have both means and motive is enough to push P(H) up to a high value.

Sure, the lack of observed sabotage (~E) makes P(H|~E) < P(H). So the probability of a fifth column goes down a bit. But P(H) started out so high that H is still the only contingency that we should really worry about. The only important question left is, Given that there is a fifth column, is it competent or incompetent? Does the observation of ~E mean that we are in more danger or less danger? That is, letting C = "The fifth column is competent", do we have that P(C | ~E & H) > P(C | H)?

Warren is arguing that ~E should lead us to anticipate a more dangerous fifth column. He is saying that an incompetent fifth column would probably have performed minor sabotage, which would have left evidence. A competent fifth column, on the other hand, would probably still be marshaling its forces to strike a major blow, which would be inconsistent with E. Hence, P(C | ~E & H) > P(C | H). That is why ~E is a greater cause for concern than E would have been.

Whether all of these prior probabilities are reasonable is another matter. But Warren's remarks are consistent with correct Bayesian reasoning from those priors.

Comment author: bigjeff5 29 January 2011 01:21:46AM -1 points [-]

Warren stated in the quote that the lack of any subversive activity was the most convincing factor of all the evidence he has that the 5th Column would soon commit subversive activity.

The problem here should be pretty obvious.

As soon as any subversive activity occurs, the evidence that the 5th Column is going to commit subversive activity clearly just went down! And since the lack of evidence was the strongest evidence for this fact, the fact that the "lack of evidence" is now 0 (either evidence exists, or no evidence exists, there are no degrees for this type of evidence) makes it impossible for the 5th Column to have committed the subversive activity!

The absurdity of this reasoning should be obvious, and it should be thrown out immediately. The lack of subversive activity was clearly not evidence that the 5th Column was planning something. It could not be. You might think the 5th Column was planning something based on other evidence, and that is perfectly fine, but your reasoning for the risk of a subversive activity cannot be based on the lack of any subversive activity. It must be based on other evidence or it invalidates itself.

Comment author: Tyrrell_McAllister 30 January 2011 01:14:24AM * 9 points [-]

Warren stated in the quote that the lack of any subversive activity was the most convincing factor of all the evidence he has that the 5th Column would soon commit subversive activity.

(Emphasis added.)

I just don't see that in the quote. Here is the Warren quote from the OP:

"I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed... I believe we are just being lulled into a false sense of security."

His claim isn't that subversive activity will start soon. The claim is that subversive activity will be "timed just like Pearl Harbor was timed". I read this to mean that he anticipates a centrally-orchestrated, synchronized, large-scale attack, of the sort that could only be pulled off by a disciplined, highly-competent fifth column.

If they had seen small, piece-meal efforts at sabotage, then that would have been evidence against a competent fifth column. That is, P(there is a competent fifth column | there has been piece-meal sabotage) < P(there is a competent fifth column).

Therefore, not seeing such efforts is evidence for a competent fifth column: P(there is a competent fifth column | there has been no piece-meal sabotage) > P(there is a competent fifth column). This is a direct algebraic consequence of Bayes's formula.

Of course, seeing no piece-meal sabotage is also evidence for there being no fifth column at all. But if your prior for "no fifth column" is sufficiently low, it still makes sense to spend most of your effort on interpreting what the no-sabotage evidence says about the nature of the fifth column, given that it exists. And what it says, given that there is a fifth column, is that the fifth column is probably marshaling its forces to strike a major blow. (Or at least, that's what the no-sabotage evidence says under the right prior.)

As soon as any subversive activity occurs, the evidence that the 5th Column is going to commit subversive activity clearly just went down!

Scattered and piecemeal acts of sabotage would show that the fifth column is incompetent. So such activity would make "our situation" less "ominous". This is consistent with Warren's view. Such sabotage wouldn't make the probability of subversive activity go down, but Warren doesn't say that it would. But such sabotage would make the probability of sabotage comparable to Pearl Harbor go down. That is Warren's claim.

And since the lack of evidence was the strongest evidence for this fact, the fact that the "lack of evidence" is now 0 (either evidence exists, or no evidence exists, there are no degrees for this type of evidence) ...

This is where we disagree. It's not a matter of "sabotage" vs. "no sabotage". Incompetent sabotage is different from competent sabotage. Warren has a prior that assigns a high prior probability to the existence of a fifth column. His priors about how fifth-columns work, as a function of their competence, are evidently such that our significantly-probable states, in increasing order of ominousness, are

having seen incompetent sabotage,
having seen no sabotage yet,
having seen competent sabotage.

Warren believed that we were in the middle state.

The absurdity of this reasoning should be obvious, and it should be thrown out immediately. The lack of subversive activity was clearly not evidence that the 5th Column was planning something. It could not be. You might think the 5th Column was planning something based on other evidence, and that is perfectly fine, but your reasoning for the risk of a subversive activity cannot be based on the lack of any subversive activity. It must be based on other evidence or it invalidates itself.

In my previous comment, I gave a Bayesian explanation of how the lack of subversive activity could be evidence that we are in a more dangerous situation than we would have been if we had seen evidence of subversion. That is, given the right priors, the lack of subversive activity could be "ominous". Can you point to an error in my reasoning?

Comment author: bigjeff5 comment score below threshold [+] (15 children)

Comment author: Tyrrell_McAllister 30 January 2011 05:14:32AM * 3 points [-]

Of course, seeing no piece-meal sabotage is also evidence for there being no fifth column at all.

That makes no sense at all. How can a fact be both for and against the same thing? You can't split the evidence. You can't say 20% of the time it means there isn't a 5th Column and 80% of the time it means there is.

It is possible to have two mutually exclusive propositions, Q and R, such that some observation E is both evidence for Q and evidence for R. That is, it is possible to have both p(Q|E) > p(Q) and p(R|E) > p(R), even though Q implies not-R.

Do you disagree?

In my argument above,

Q is "there is a competent fifth column",
R is "there is no fifth column", and
E is "there has been no piece-meal sabotage".

Before moving on to the other issues, does this part make sense now?

ETA: It's important to note that Q and R, while mutually exclusive, are not exhaustive. A third possibility is that there is fifth column, but it's incompetent. Q is not equivalent to not-R.

Comment author: bigjeff5 30 January 2011 07:57:59PM 0 points [-]

Warren did not say anything about piece-meal sabotage. He called it a "lack of subversive activity", and he didn't put limits on what type of subversion.

The way you described E is also counter-intuitive and confusing, and is not the way Warren described it. E must be that there has been subversive activity, that's what Warren described. He said the lack of E (~E) was evidence of Q (p(Q|~E)). That's fine, as long as E must be evidence that Q is not true (p(~Q|E)).

Since ~E is weak evidence for Q, E is strong evidence for ~Q. The next subversive activity of any kind will be the first instance of E, and it will invalidate his entire reasoning for detaining the Japanese-American citizens.

Comment author: Tyrrell_McAllister 30 January 2011 09:06:53PM * 2 points [-]

Warren did not say anything about piece-meal sabotage. He called it a "lack of subversive activity", and he didn't put limits on what type of subversion.

The way you described E is also counter-intuitive and confusing, and is not the way Warren described it.

Okay, replace my earlier definition of E with

E is "we have seen no subversive activity".

Do you agree that, under some priors, you could have p(Q|E) > p(Q) and p(R|E) > p(R), even though Q implies not-R?

Set aside the question of whether these are reasonable priors. My point was only this; Warren didn't make the simple mistake with the probability calculus that Eliezer thought Warren made. He wasn't simultaneously asserting p(H|E) > p(H) and p(H|~E) > p(H). That would be wrong under any prior, no matter how bizarre. But it's not what Warren was doing.

What Warren said is consistent with coherent Bayesian updating, even if he was updating on a bizarre prior. It might have been wrong to put a high prior probability on subversive activity, but the probability calculus doesn't tell you how to pick your prior. All I am saying is that the Warren quote, in and of itself, does not constitute a violation of the rules of the probability calculus.

Maybe Warren committed such a violation earlier on. Maybe that's how he arrived at such a high prior for the existence of subversive activity. But those earlier steps in his reasoning aren't laid out before us here, so we can't point to any specific misapplication of Bayes's rule, as Eliezer tried to do.

Comment author: bigjeff5 31 January 2011 12:55:22AM * 0 points [-]

E is "we have seen no subversive activity".

I don't like the way you describe that. It is confusing. The evidence is subversive activity. You cannot go out and look for no subversive activity, that makes no sense. You have to look for subversive activity. I'm not sure why you're fighting so hard for this point, since not finding something suggests just as much as finding something does. The only reason I suggest a change is for clarity. I don't want to think about no subversive activity and not no subversive activity, I want to think about subversive activity or no subversive activity. There is no difference, the second is simply less confusing.

E is subversive activity, and Warren's position is p(Q|~E).

Do you agree that, under some priors, you could have p(Q|E) > p(Q) and p(R|E) > p(R), even though Q implies not-R?

Absolutely. I said so two posts up. The question is not about Q and R, though, it's about Q and ~Q.

Set aside the question of whether these are reasonable priors.

But the whole argument is about the priors. The reason Warren's position is nonsensical is not because he believes a lack of subversion suggests some fact, it's that he argues that a lack of subversion suggests a fact, and then behaves in a manner counter to his argument. I've been arguing the fact that Warren argues p(Q|~E), But the reason he is locking up the Japanese-Americans is because he expects p(E|Q). The only way p(E|Q) makes any sense is if p(Q|E) is also true.

Warren's fundamental fear is based on p(E|Q) - that is, the 5th Column is plotting and scheming, and this will lead to subversion. The argument he uses to support this, however, is that p(Q|~E). The two positions are inversely related. If p(E|Q) is strong, then p(Q|~E) must be weak.

In other words, if p(E|Q) is strong, and p(Q) is high, then p(E) should be very high (because Q implies E), and p(~E) should be very small. Yet a very high ~E is used as evidence of p(Q). That makes no sense. If p(E|Q) is high, then ~E can exist in spite of Q, but it cannot exist because of Q.

The only way this is at all tenable is if p(E|Q) and p(~E|Q) are both true. In which case, neither E nor ~E is evidence of Q.

That's the whole point.

probability calculus doesn't tell you how to pick your prior.

The whole point of this discussion is that his reasoning does not coincide with his actions. Thus one or the other is wrong.

Comment author: CuSithBell 31 January 2011 04:09:00AM 2 points [-]

Doesn't his position make sense if he believes that:

if there's no organized fifth column, we should see some intermittent, disorganized sabotage, and
if there is an organized fifth column, we should see NO sabotage before some date, at which there is a devastating attack

Of course, I agree that it's likely he would have made a different argument if he had seen evidence of sabotage - but as presented it seems his position is at least potentially coherent.

Comment author: bigjeff5 31 January 2011 05:32:39PM -1 points [-]

But what is the date? Is it 2 months? 6 months? A year? 5 years? What if it never happens? If nothing happens in 200 years, does that mean we must be absolutely certain that a fifth column is planning an attack?

It's not evidence, it's a lack of evidence. That's the point, and that's the problem.

Warren states it is his most ominous evidence that they are planning something. What evidence? It's not there, it doesn't exist. His whole position is based on the idea that the lack of evidence indicates they are planning something, yet he has nothing to suggest that such a lack of evidence indicates anything. The only thing the fact that they haven't attacked yet is evidence for is that they haven't attacked yet. Nothing more, nothing less, unless you have a pattern of behavior to base that on. There was no such pattern for the fifth column.

continue this thread »

Comment author: Tyrrell_McAllister 31 January 2011 06:00:30AM * 1 point [-]

E is "we have seen no subversive activity".

I don't like the way you describe that. It is confusing. The evidence is subversive activity. You cannot go out and look for no subversive activity, that makes no sense.

Our evidence is always only what we have observed. Maybe it is strange to say that you "looked for no subversive activity". But you certainly can look for subversive activity and fail to find it. Not seeing subversive activity when you looked for it is Bayesian evidence. But it would be an error to condition on there being no subversive activity at all, even hidden activity. That would be going beyond your observations. You can only condition on what you saw or didn't see when you looked.

Do you agree that, under some priors, you could have p(Q|E) > p(Q) and p(R|E) > p(R), even though Q implies not-R?

Absolutely. I said so two posts up. The question is not about Q and R, though, it's about Q and ~Q.

Okay, I think that we're homing in on the nub of the disagreement.

The propositions in question are

"There is a fifth column that will coordinate a Pearl-Harbor type attack", and
"There is no fifth column".

These are Q and R, respectively. They are not negations of each other. Do you agree?

Comment author: bigjeff5 31 January 2011 05:04:55PM 0 points [-]

Again, I cannot see how you can observe nothing and call it evidence. It is semantics, really, since it makes no difference for the equations, but it makes ~E a positive observation of something and E a negative observation, which is, to me, silly.

These are Q and R, respectively. They are not negations of each other. Do you agree?

Yes. Though, again, I'd rather R be "There is a 5th column" to keep it from being confusing.

With Q there was no evidence that the fifth column was coordinating a timed attack, yet Warren's strongest evidence for it was that there was no evidence for it.

Pearl harbor types of evidence are black swans. You can't just pull them out of the air and add them to your reasoning when you have no solid justification for it. There are a billion other black swans he could have used - what if the Japanese are actually all vampires and had designs on draining the Americans dry? You've got no evidence they aren't, so clearly they are just biding their time!

The former is slightly more reasonable, since something similar had happened recently (though in an entirely different context), but it is no more justified as evidence than the evidence in the vampire scenario.

You must look for other evidence that suggests the 5th column was planning an attack, the fact that you have not been attacked yet is not in any way evidence that they are planning an attack. It is only really evidence that, if they were planning something, they hadn't done it yet. That's all you can get from that - just a guess.

To that end, Warren had no evidence that an American chapter of the 5th Column even existed. There was secret evidence to that effect, but he was not privy to it. He was making the whole thing up because he was afraid.

It was completely unjustified.

Besides, it doesn't make sense. Timed attacks are designed to catch you off guard. After Pearl Harbor, people were always on guard. It wouldn't have had the same effect; a much more effective strategy would have been smaller, guerrilla-type sabotages from within, which they also had zero evidence of.

continue this thread »

Comment author: thomblake 31 January 2011 09:26:33PM 0 points [-]

While I think your reading is consistent with a very generous application of the principle of charity, I'm not certain it's appropriate in this case to so apply. Do you have any evidence that Warren was reasoning in this way rather than the less-charitable version, and if so, why didn't he say so explicitly?

It really seems like the simpler explanation is fear plus poor thinking.

Comment author: Tyrrell_McAllister 23 March 2011 04:20:30PM * 1 point [-]

Do you have any evidence that Warren was reasoning in this way rather than the less-charitable version, and if so, why didn't he say so explicitly

Sorry for taking so long to reply to this.

I think that a close and strict reading supports my interpretation. I don't see the need for an unduly charitable reading.

First, I assume the following context for the quote: Warren had argued for (or maybe only claimed) a high probability for the proposition that there is a Japanese fifth column within the US. Let R be this italicized proposition. Then Warren has argued that p(R) >> 0.

Given that context, here is how I parse the quote, line-by-line:

[A] questioner pointed out that there had been no sabotage or any other type of espionage by the Japanese-Americans up to that time.

I take the questioner to be asserting that there has been no observed sabotage or any other type of espionage by Japanese-Americans up to that time. Let E be this proposition.

Warren responds:

I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation.

I take Warren to be saying that the expected cost of not interring Japanese-Americans is significantly higher after we update on E than it was before we updated on E. Letting D be the "default" action in which we don't inter Japanese-Americans, Warren is asserting that EU(D | E) << EU(D).

The above assertion is the conclusion of Warren's reasoning. If we can show that this conclusion follows from correct Bayesian reasoning from a psychologically realistic prior, plus whatever evidence he explicitly adduces, then the quote cannot serve as an example of the fallacy that Eliezer describes in this post.

Now, we may think that that "psychologically realistic prior" is very probably based in turn on "fear plus poor thinking". But Warren doesn't explicitly show us where his prior came from, so the quote in and of itself is not an example of an explicit error in Bayesian reasoning. Whatever fallacious reasoning occurred, it happened "behind the scenes", prior to the reasoning on display in the quote.

Continuing with my parsing, Warren goes on to say:

It [this lack of subversive activity] convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed... I believe we are just being lulled into a false sense of security.

Let Q be the proposition that there is a Japanese fifth column in America, and it will perform a timed attack, but right now it is lulling us into a false sense of security.

I take Warren to be claiming that p(Q | E) >> p(Q), and that p(Q | E) is sufficiently large to justify saying "I believe Q".

It remains only to give a psychologically realistic prior distribution p such the claims above follow — that is, we need that

p(R) is sufficiently large to justify saying "R has high probability",
p(E) = 1 - epsilon,
EU(D | E) << EU(D),
p(Q | E) >> p(Q),
p(Q | E) is sufficiently large to justify saying "I believe Q".

This will suffice to invalidate the Warren quote as an example for this post.

It is a mathematical fact that such priors exist in an abstract sense. Do you think it unlikely that such a prior is psychologically realistic for someone in Warren's position? I think that selection effects and standard human biases make it very plausible that someone in his position would have such a prior.

If you're still skeptical, we can discuss which priors are psychologically realistic for someone in Warren's position.

Tyrrell_McAllister comments on Absence of Evidence Is Evidence of Absence - Less Wrong

Article Navigation

Comments (108)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (108)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login