David Chapman criticizes "pop Bayesianism" as just common-sense rationality dressed up as intimidating math[1]:
Bayesianism boils down to “don’t be so sure of your beliefs; be less sure when you see contradictory evidence.”Now that is just common sense. Why does anyone need to be told this? And how does [Bayes'] formula help?
[...]
The leaders of the movement presumably do understand probability. But I’m wondering whether they simply use Bayes’ formula to intimidate lesser minds into accepting “don’t be so sure of your beliefs.” (In which case, Bayesianism is not about Bayes’ Rule, after all.)
I don’t think I’d approve of that. “Don’t be so sure” is a valuable lesson, but I’d rather teach it in a way people can understand, rather than by invoking a Holy Mystery.
What does Bayes's formula have to teach us about how to do epistemology, beyond obvious things like "never be absolutely certain; update your credences when you see new evidence"?
I list below some of the specific things that I learned from Bayesianism. Some of these are examples of mistakes I'd made that Bayesianism corrected. Others are things that I just hadn't thought about explicitly before encountering Bayesianism, but which now seem important to me.
I'm interested in hearing what other people here would put on their own lists of things Bayesianism taught them. (Different people would make different lists, depending on how they had already thought about epistemology when they first encountered "pop Bayesianism".)
I'm interested especially in those lessons that you think followed more-or-less directly from taking Bayesianism seriously as a normative epistemology (plus maybe the idea of making decisions based on expected utility). The LW memeplex contains many other valuable lessons (e.g., avoid the mind-projection fallacy, be mindful of inferential gaps, the MW interpretation of QM has a lot going for it, decision theory should take into account "logical causation", etc.). However, these seem further afield or more speculative than what I think of as "bare-bones Bayesianism".
So, without further ado, here are some things that Bayesianism taught me.
- Banish talk like "There is absolutely no evidence for that belief". P(E | H) > P(E) if and only if P(H | E) > P(H). The fact that there are myths about Zeus is evidence that Zeus exists. Zeus's existing would make it more likely for myths about him to arise, so the arising of myths about him must make it more likely that he exists. A related mistake I made was to be impressed by the cleverness of the aphorism "The plural of 'anecdote' is not 'data'." There may be a helpful distinction between scientific evidence and Bayesian evidence. But anecdotal evidence is evidence, and it ought to sway my beliefs.
- Banish talk like "I don't know anything about that". See the post "I don't know."
- Banish talk of "thresholds of belief". Probabilities go up or down, but there is no magic threshold beyond which they change qualitatively into "knowledge". I used to make the mistake of saying things like, "I'm not absolutely certain that atheism is true, but it is my working hypothesis. I'm confident enough to act as though it's true." I assign a certain probability to atheism, which is less than 1.0. I ought to act as though I am just that confident, and no more. I should never just assume that I am in the possible world that I think is most likely, even if I think that that possible world is overwhelmingly likely. (However, perhaps I could be so confident that my behavior would not be practically discernible from absolute confidence.)
- Absence of evidence is evidence of absence. P(H | E) > P(H) if and only if P(H | ~E) < P(H). Absence of evidence may be very weak evidence of absence, but it is evidence nonetheless. (However, you may not be entitled to a particular kind of evidence.)
- Many bits of "common sense" rationality can be precisely stated and easily proved within the austere framework of Bayesian probability. As noted by Jaynes in Probability Theory: The Logic of Science, "[P]robability theory as extended logic reproduces many aspects of human mental activity, sometimes in surprising and even disturbing detail." While these things might be "common knowledge", the fact that they are readily deducible from a few simple premises is significant. Here are some examples:
- It is possible for the opinions of different people to diverge after they rationally update on the same evidence. Jaynes discusses this phenomenon in Section 5.3 of PT:TLoS.
- Popper's falsification criterion, and other Popperian principles of "good explanation", such as that good explanations should be "hard to vary", follow from Bayes's formula. Eliezer discusses this in An Intuitive Explanation of Bayes' Theorem and A Technical Explanation of Technical Explanation.
- Occam's razor. This can be formalized using Solomonoff induction. (However, perhaps this shouldn't be on my list, because Solomonoff induction goes beyond just Bayes's formula. It also has several problems.)
- You cannot expect[2] that future evidence will sway you in a particular direction. "For every expectation of evidence, there is an equal and opposite expectation of counterevidence."
- Abandon all the meta-epistemological intuitions about the concept of knowledge on which Gettier-style paradoxes rely. Keep track of how confident your beliefs are when you update on the evidence. Keep track of the extent to which other people's beliefs are good evidence for what they believe. Don't worry about whether, in addition, these beliefs qualify as "knowledge".
What items would you put on your list?
ETA:
[1] See also Yvain's reaction to David Chapman's criticisms.
[2] ETA: My wording here is potentially misleading. See this comment thread.
Lumifer, you are falling prey to several of the traps detailed in A Human's Guide to Words. So far I have basically parroted EY's 102 material.
Meditation: Taboo "Knowledge" and describe your relation with riding a bicycle.
Meditation: Taboo "Knowledge" and describe your relation with some field of science you are proficient in.
Meditation: Taboo "Knowledge" and describe a religious person's views on god.
...
...
You and me both know what 'knowledge' is in everyday speech. The problem is what constitues 'knowlege' in extreme situations.
The thing is that "Knowledge" is ambiguous in everyday speech. We misunderstood each other when I initially answered your question: I thought you were speaking about the tried and tired philosophical issue that have been discussed for ages.
The answer in the Philosophical Issue of Knowledge is: "You philosophers are all morons; you are using the same word to mean different things."
Plato has a famous definition of "Knowledge": Justified True Belief. Notice how he has moved the problem of explaining "Knowledge" into the problem of explaining "Justification." (And "True." And "Belief." Neither concepts were actually well explained when Plato was alive and kicking.)
"Knowledge" can also be a synonym for "Skill." Such as knowing how to ride a bicycle. Notice how the grammatical construction "knowing how to ." is different from "knowing to be true." One could argue that they are the same thing, but I think they are not. So we have at least two types of everyday discussed knowledge: Procedural Knowledge (how to do stuff) and Object Knowledge (facts and stuff).
The distinction between the two is obvious when you really taboo it: Procedural knowledge is like a tool. It is a means to an end, an extension of your primitive action set. Having lots of procedural knowledge is a boon in Instrumental Rationality, but most skills are irrelevant to Epistemological Rationality. (Riding a cicycle will only very rarely tell you the secrets of the universe.)
Object Knowledge, or Facts, are thingies in your mental model of how the world works. This mental model is what you use when you want to predict how the world is going to behave in the future, so that you can make plans. (Because you have goals you want to attain.)
Your world model is updated automatically by processes which you do not control. A sufficiently advanced agent might be able to excercise some control, at least at the design level, of it's updating algorithms. In short, you take in sensory data and crunch some numbers and out comes a bayesian-esque update.
So my standing viewpoint is: I don't care what you call it; "knowledge" or "hunch" or "divine inspiration." I care about what your probability distribution over future events is. I don't care what you call it "skills" or "knowledge" or "talent." I care about what sort of planning algorithm you implement.
And on the topic of subjectivity: If I have trained skills or observed evidence different from you, then yes we have subjectively different "knowledge." I for instance know 12 programming languages and intimate facts about my significant other.
But the thing is that there is only One Correct Way of updating on evidence: Bayes Theorem. If you deviate from that you will have less than optimal predictive power.
I really suggest you go and read some of the core sequences to refresh this.
I think the dichotomy between procedural knowledge and object knowledge is overblown, at least in the area of science. Scientific object knowledge is (or at least should be) procedural knowledge: it should enable you to A) predict what will happen in a given situation (e.g. if someone drops a mento into a bottle of diet coke) and B) predict how to set up a situation to achieve a desired result (e.g. produce pure L-glucose).