Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Fallacies as weak Bayesian evidence

59 Kaj_Sotala 18 March 2012 03:53AM

Abstract: Exactly what is fallacious about a claim like ”ghosts exist because no one has proved that they do not”? And why does a claim with the same logical structure, such as ”this drug is safe because we have no evidence that it is not”, seem more plausible? Looking at various fallacies – the argument from ignorance, circular arguments, and the slippery slope argument - we find that they can be analyzed in Bayesian terms, and that people are generally more convinced by arguments which provide greater Bayesian evidence. Arguments which provide only weak evidence, though often evidence nonetheless, are considered fallacies.

As a Nefarious Scientist, Dr. Zany is often teleconferencing with other Nefarious Scientists. Negotiations about things such as ”when we have taken over the world, who's the lucky bastard who gets to rule over Antarctica” will often turn tense and stressful. Dr. Zany knows that stress makes it harder to evaluate arguments logically. To make things easier, he would like to build a software tool that would monitor the conversations and automatically flag any fallacious claims as such. That way, if he's too stressed out to realize that an argument offered by one of his colleagues is actually wrong, the software will work as backup to warn him.

Unfortunately, it's not easy to define what counts as a fallacy. At first, Dr. Zany tried looking at the logical form of various claims. An early example that he considered was ”ghosts exist because no one has proved that they do not”, which felt clearly wrong, an instance of the argument from ignorance. But when he programmed his software to warn him about sentences like that, it ended up flagging the claim ”this drug is safe, because we have no evidence that it is not”. Hmm. That claim felt somewhat weak, but it didn't feel obviously wrong the way that the ghost argument did. Yet they shared the same structure. What was the difference?

The argument from ignorance

Related posts: Absence of Evidence is Evidence of Absence, But Somebody Would Have Noticed!

One kind of argument from ignorance is based on negative evidence. It assumes that if the hypothesis of interest were true, then experiments made to test it would show positive results. If a drug were toxic, tests of toxicity of reveal this. Whether or not this argument is valid depends on whether the tests would indeed show positive results, and with what probability.

With some thought and help from AS-01, Dr. Zany identified three intuitions about this kind of reasoning.

1. Prior beliefs influence whether or not the argument is accepted.

A) I've often drunk alcohol, and never gotten drunk. Therefore alcohol doesn't cause intoxication.

B) I've often taken Acme Flu Medicine, and never gotten any side effects. Therefore Acme Flu Medicine doesn't cause any side effects.

Both of these are examples of the argument from ignorance, and both seem fallacious. But B seems much more compelling than A, since we know that alcohol causes intoxication, while we also know that not all kinds of medicine have side effects.

2. The more evidence found that is compatible with the conclusions of these arguments, the more acceptable they seem to be.

C) Acme Flu Medicine is not toxic because no toxic effects were observed in 50 tests.

D) Acme Flu Medicine is not toxic because no toxic effects were observed in 1 test.

C seems more compelling than D.

3. Negative arguments are acceptable, but they are generally less acceptable than positive arguments.

E) Acme Flu Medicine is toxic because a toxic effect was observed (positive argument)

F) Acme Flu Medicine is not toxic because no toxic effect was observed (negative argument, the argument from ignorance)

Argument E seems more convincing than argument F, but F is somewhat convincing as well.

"Aha!" Dr. Zany exclaims. "These three intuitions share a common origin! They bear the signatures of Bayonet reasoning!"

"Bayesian reasoning", AS-01 politely corrects.

"Yes, Bayesian! But, hmm. Exactly how are they Bayesian?"

continue reading »

Lists of cognitive biases, common misconceptions, and fallacies

13 Kevin 23 January 2010 10:47PM





I know the trend here is against simple links as top-level posts, but I think these links are powerful enough to stand on their own. I would be very surprised if most people here have already read all of these links. Also, it seems like a good sign that this somehow got modded up to 1 while sitting in my drafts folder.

Thanks to Lone Gunman for the links.

Defense Against The Dark Arts: Case Study #1

101 Yvain 28 March 2009 02:31AM

Related to: The Power of Positivist Thinking, On Seeking a Shortening of the Way, Crowley on Religious Experience

Annoyance wants us to stop talking about fancy techniques and get back to basics. I disagree with the philosophy behind his statement, but the principle is sound. In many areas of life - I'm thinking mostly of sports, but not for lack of alternatives - mastery of the basics beats poorly-grounded fancy techniques every time.

One basic of rationality is paying close attention to an argument. Dissecting it to avoid rhetorical tricks, hidden fallacies, and other Dark Arts.  I've been working on this for years, and I still fall short on a regular basis.

Medical educators have started emphasizing case studies in their curricula. Instead of studying arcane principles of disease, student doctors cooperate to analyze a particular patient in detail, ennumerate the principles needed to diagnose her illness, and pay special attention to any errors the patients' doctors made during the treatment. The cases may be rare tropical infections, but they're more often the same everyday diseases common in the general population, forcing the student doctors to always keep the basics in mind. We could do with a tradition of case studies in rationality, though we'd need safeguards to prevent degeneration into political discussion.

Case studies in medicine are most interesting when all the student doctors disagree with each other. To that end, I've chosen as the first case a statement that received sixteen upvotes on Less Wrong, maybe the highest I've ever seen for a comment. I don't mean to insult or embarass everyone who liked it. I liked it too. My cursor was already hovering above the "Vote Up" button by the time I starting having second thoughts. But it deserves dissection, and its popularity gives me a ready response when someone says this material is too basic for 'master rationalists' like ourselves:

In his youth, Steve Jobs went to India to be enlightened. After seeing that the nation claiming to be the source of this great spiritual knowledge was full of hunger, ignorance, squalor, poverty, prejudice, and disease, he came back and said that the East should look to the West for enlightenment.

This anecdote is short, witty, flattering, and utterly opaque to reason. It bears all the hallmarks of the Dark Arts.

continue reading »

The Third Alternative

55 Eliezer_Yudkowsky 06 May 2007 11:47PM

"Believing in Santa Claus gives children a sense of wonder and encourages them to behave well in hope of receiving presents.  If Santa-belief is destroyed by truth, the children will lose their sense of wonder and stop behaving nicely.  Therefore, even though Santa-belief is false-to-fact, it is a Noble Lie whose net benefit should be preserved for utilitarian reasons."

Classically, this is known as a false dilemma, the fallacy of the excluded middle, or the package-deal fallacy.  Even if we accept the underlying factual and moral premises of the above argument, it does not carry through.  Even supposing that the Santa policy (encourage children to believe in Santa Claus) is better than the null policy (do nothing), it does not follow that Santa-ism is the best of all possible alternatives.  Other policies could also supply children with a sense of wonder, such as taking them to watch a Space Shuttle launch or supplying them with science fiction novels.  Likewise (if I recall correctly), offering children bribes for good behavior encourages the children to behave well only when adults are watching, while praise without bribes leads to unconditional good behavior.

Noble Lies are generally package-deal fallacies; and the response to a package-deal fallacy is that if we really need the supposed gain, we can construct a Third Alternative for getting it.

continue reading »

View more: Next