You talk about real evidence, but you don't unpack "real". For my money, the crucial point is whether you made an active effort to find evidence. It's rather vacuous to say there is no evidence, meaning that it has not landed on your doorstep. And it's a great way of supporting biases.
The idea that you need to actively seek confirmation, and, much more importantly, refutation is central to the Popperian approach. The Bayesian approach seems strangely passive, by contrast.
In a Bayesian context, seeking evidence is about narrowing the probability distribution from what should be a relatively flat prior. One could probably make a case for not making a decision until the cost of putting it off outweighs the gain by decreasing the uncertainty.
"Narrowing" can be passive or active. Passive narrowing is going to be a lot less efficient than active narrowing. If you are in a position to wait for as long as it takes , that would not be a problem...but you aren't't. You also don't have a pre existing mental database of every possible hypothesis, or the ability to assign infinitesimal probabilities to them.
If we treat the “is” in Absence of Evidence is Evidence of Absence as an “implies” (which it seems to me to be) and then apply modus tollens to it, we get “if you don’t have evidence of absence, you don’t have absence of evidence” and it is precisely this bullshit that Zvi is calling. If you have evidence of absence, say so.
Two comments:
First, as Jiro said, "implies" replaces "is evidence of", not just "is".
But second, since this is a probabilistic statement, using logical "implies" and modus tollens isn't appropriate.
So it would be "Absence of Evidence suggests Absence" (absence makes absence of evidence more likely), and the contrapositive would be "Presence Suggests Evidence" (presence makes evidence more likely).
But it is not just any absence of evidence, it is absence of evidence when we expect to see evidence in the case where the hypothesis is true. The full statement would be: Absence of evidence is evidence of absence proportional to expected evidence given presence. If we wouldn't expect any evidence for some hypothesis even if it was true, then an absence of evidence wouldn't be any evidence of absence.
The reason "no evidence" claims are often fallacious (even when they don't ignore evidence, as the original post shows they often do), is because they tend to treat absence of evidence as evidence of absence when we expect an absence of evidence (e.g, because we didn't have time to collect evidence yet) and therefore shouldn't actually update on it.
I do think you're getting it correct in the There is no No Evidence and Exactly what evidence isn't there? sections.
I'm surprised to see this downvoted. This comment follows all the discussion norms. What about this comment would you "like to see less of"? If you think there's a mistake here, explain it, I'd like to know. (ETA: the first vote on the parent was a downvoted, and it remained the only vote for about a day)
If we treat the “is” in Absence of Evidence is Evidence of Absence as an “implies” (which it seems to me to be) and then apply modus tollens to it, we get “if you don’t have evidence of absence, you don’t have absence of evidence” and it is precisely this bullshit that Zvi is calling. If you have evidence of absence, say so.
No. The "is" in that doesn't mean "implies". The "is evidence of" means "implies".
Modus tollens then gives you "if you don't have absence, you don't have absence of evidence", which is not subject to the semantic tricks in the post.
It's important to distinguish between different kinds of evidence. There's never no rational evidence, but there might be no scientific evidence. The issue comes in when people conclude that there's no rational evidence because there's no scientific evidence.
The problem with the "no scientific evidence" line of thought is it devolves to:
"We know nothing at all unless a credentialed scientist conducted a blind RCT at the cost of millions of dollars and years of time. And the only information we learned from the RCT was a probability update for a binary question. And the results had to be reviewed by peer scientists, manually, and then published in a high impact journal or they are not credible".
Otherwise we are going to pretend we know absolutely nothing and will continue to do things per decades old tradition. (Even though those traditions were never checked with the same algorithm)
This is not a system that will develop effective treatments for aging and human disease.
Moreover the math says it is morally evil and kills millions.
My proposed fix is we develop automated systems to do all the above and we have the algorithms reviewed by all of the above but then reuse the same automated systems for thousands of experiments. And instead of explicit rcts we usually find the data by mining or proxy experiments.
I definitely share your concern that evidence which isn't "scientific" matters, but I still think whether or not there is scientific evidence isn't entirely irrelevant to decision-making when we care about creating organizations that consistently make good decisions.
Currently, we definitely care far too much about scientific evidence, but I disagree that the concept is entirely bullshit.
I am not saying it is bullshit. But failing to consider information also has a cost. And for some fields, "consistently good decisions" may not even be possible.
Yes, but I think making the distinction that way is going to be much harder for many people outside this community. I know very few people who don't read this website who even have any sort of probabilistic conception of "evidence" and "belief," and I have had (very conventionally well educated in STEM fields) people get angry at me for talking about things that way.
We are familiar with the ideas of rational evidence, scientific evidence, legal evidence, and so on, screening off some types of evidence for certain purposes, but most people aren't, at least not explicitly, and as far as I can tell have no idea that that's what our societal institutions are doing or why.
What's "scientific evidence"?
Things you would refer to when writing a paper about the subject? When it comes to a question like whether COVID is airbone I doubt that there's no prior art that you would cite that point into the direction.
Eliezer isn't given an exact definition but it seems to me like he's pointing to scientific knowledge being knowledge that comes from scientific papers.
There's weak evidence for most scientific statements that you could make that comes from scientific papers.
The fact that most published papers about homeopathy find that homeopathy is effective is a type of scientific evidence. On the other hand the fact that most high quality trials that are published show that it has no effect suggests that overall we should believe that it has no effect. Especially, if we combine the evidence from the high quality trials with evidence we have about published physical theories.
I can't think of good examples that are revelant for this discussion where there's no scientific evidence (nothing is published in a paper that's Bayesian evidence).
I've gotten pretty exasperated by the "there is no evidence" issue in public health communication in the last year or so, and hence appreciate this post.
Side note: There's a screenshot of a twitter conversation in this post, which doesn't load on lesswrong.com, but does load when I copy the screenshot's URL into a new window.
Zvi recently coined, and has now written up this Law of No Evidence:
Considered next to Eliezer's Absence of Evidence Is Evidence of Absence, it might seem like a contradiction, but as far as I can tell, it actually follows directly.
If we treat the "is" in Absence of Evidence is Evidence of Absence as an "implies" (which it seems to me to be) and then apply modus tollens to it, we get "if you don't have evidence of absence, you don't have absence of evidence" and it is precisely this bullshit that Zvi is calling. If you have evidence of absence, say so.
The Very Serious covid spokespeople spouting bullshit like "no evidence of human-to-human transmission" are doing a sort of naive equivocation version of "absence of evidence is evidence of absence", by saying "no evidence for" as if that implies "evidence against" when in fact they're trying to support some agenda or worldview when there's not very much clear evidence in any direction and someone could just as easily say "there's no evidence there isn't human-to-human transmission". At least, that's in the best case. Sometimes the evidence does favor a particular direction, they just don't like it and don't want to count it. Desperately clinging to priors?
Hm.
So Zvi's Law of No Evidence can be seen as taking Absence of Evidence is Evidence of Absence a step further and instead saying "if your absence of evidence is real, then it'll actually be evidence of absence, in which case call it that. otherwise shut up." But there's still a point to be made about a sense in which "absence of evidence" is its own thing—it's just different from no evidence. The important piece is that "no evidence" is bullshit, but "we didn't see this particular thing we would more expect to see if X were true than if X weren't true" is a vital component of successful reasoning about the world. It's the very basis of Bayes.
There is no No Evidence
If you observe {not[evidence of X]} then it makes sense to update towards {not[X]}, but the speech act of claiming "There is no evidence of X" only occurs when there exists some apparent evidence that someone wants to claim doesn't count as evidence. This is a frame battle disguised as argument, since there's some evidence for almost any hypothesis you can possibly imagine.
So no, there is never no evidence.
I think there's one situation in which you can almost usefully talk about "no evidence", which is essentially "I've seen no evidence that this question is worth considering at all (given my priors) so I haven't bothered looking into it." But that's a subjective statement—you're not asserting "there is no evidence", just perhaps that you have a high bar on what would count as meaningful evidence. And even so, the fact that you're somehow talking about it at all is now at least a tiny bit of evidence that the question is worth considering, so oops.
(If you're confused about the difference between {no evidence} and "no evidence", you may want to learn more about Simulacra levels.)
Exactly what evidence isn't there?
If you want to actually talk about the "absence of evidence" that you're using as evidence of absence, you have to actually refer to the specific evidence that is missing, that you'd expect to see if the proposition were true. So you might say:
It is the absence of some specific type of evidence that is a valid argumentative tool, because it is evidence against some world in which you'd expect that evidence. This is not "no evidence" in general.
From xkcd:
The hover text adds "Not to be confused with 'making money selling this stuff to OTHER people who think it works'". There's not no evidence for these phenomena—people thinking it works is some evidence. But there's some very important missing evidence that is much stronger in the "doesn't work" direction than the "people thinking it works" is in the other direction.
There is evidence for and evidence against but there is no "no evidence".
Covid: the good and the ugly
Most official covid takes has been spectacularly bad on this front, which is of course what prompted Zvi to coin the law now, even though Overcoming Bias had Doctor, There are Two Kinds of "No Evidence" back in 2008. Here's one egregious one:
But progress is possible: a conversation I and others had with a epidemiologist on twitter managed to prompt him to actually revise his statements to avoid any "no evidence" constructions, and it turned out pretty well. Instead, he notes an absence of a specific kind of evidence (cases detected as genetically matching certain strains) and then contextualizes that to conclude that despite that lack of evidence-for-those-variants-in-the-USA, we can't actually confidently conclude said variants aren't in the USA, by offering another explanation for the absence-of-evidence other than absence-of-intended-to-be-measured-phenomenon.
There's much more work yet to be done to spread the word that "no evidence" is bullshit. This post is intended to be one place to point people for that purpose, but honestly it's written using pretty ingroupy jargony language since it started as a comment critiquing this post that made a case in favor of Zvi's Law of No Evidence but a case that I thought was confused.
I would encourage someone to write up a more generally accessible article on this!
Update: there's now Zvi's canonical Law of No Evidence post and Scott's The Phrase “No Evidence” Is a Red Flag for Bad Science Communication.