cousin_it comments on What Bayesianism taught me - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (201)
A typical situation is that there's a contentious issue, and some anecdotes reach your attention that support one of the competing hypotheses.
You have three ways to respond:
In almost every situation you're likely to encounter, the real danger is 3. Well-known biases are at work pulling you towards 3. These biases are often known to work even when you're aware of them and trying to counteract them. Moreover, the harm from reaching 3 is typically far greater than the harm from reaching 1. This is because the correct added amount of credence in 2 is very tiny, particularly because you're already likely to know that the competing hypotheses for this issue are all likely to have anecdotes going for them. In real-life situations, you don't usually hear anecdotes supporting an incredibly unlikely-seeming hypothesis which you'd otherwise be inclined to think as capable of nurturing no anecdotes at all. So forgoing that tiny amount of credence is not nearly as bad as choosing 3 and updating, typically, by a large amount.
The saying "The plural of anecdotes is not data" exists to steer you away from 3. It works to counteract the very strong biases pulling you towards 3. Its danger, you are saying, is that it pulls you towards 1 rather than the correct 2. That may be pedantically correct, but is a very poor reason to criticize the saying. Even with its help, you're almost always very likely to over-update - all it's doing is lessening the blow.
Perhaps this as an example of "things Bayesianism has taught you" that are harming your epistemic rationality?
A similar thing I noticed is disdain towards "correlation does not imply causation" from enlightened Bayesians. It is counter-productive.
Thanks for that comment! Eliezer often says people should be more sensitive to evidence, but an awful lot of real-life evidence is in fact much weaker, noisier, and easier to misinterpret than it seems. And it's not enough to just keep in mind a bunch of Bayesian mantras - you need to be aware of survivor bias, publication bias, Simpson's paradox and many other non-obvious traps, otherwise you silently go wrong and don't even know it. In a world where most published medical results fail to replicate, how much should we trust our own conclusions?
Would it be more honest to recommend people to just never update at all? But then everyone will stick to their favorite theories forever... Maybe an even better recommendation would be to watch out for "motivated cognition", try to be more skeptical of all theories including your favorites.