This article is an attempt to summarize basic material, and thus probably won't have anything new for the hard core posting crowd. It'd be interesting to know whether you think there's anything essential I missed, though.
You've probably seen the word 'Bayesian' used a lot on this site, but may be a bit uncertain of what exactly we mean by that. You may have read the intuitive explanation, but that only seems to explain a certain math formula. There's a wiki entry about "Bayesian", but that doesn't help much. And the LW usage seems different from just the "Bayesian and frequentist statistics" thing, too. As far as I can tell, there's no article explicitly defining what's meant by Bayesianism. The core ideas are sprinkled across a large amount of posts, 'Bayesian' has its own tag, but there's not a single post that explicitly comes out to make the connections and say "this is Bayesianism". So let me try to offer my definition, which boils Bayesianism down to three core tenets.
We'll start with a brief example, illustrating Bayes' theorem. Suppose you are a doctor, and a patient comes to you, complaining about a headache. Further suppose that there are two reasons for why people get headaches: they might have a brain tumor, or they might have a cold. A brain tumor always causes a headache, but exceedingly few people have a brain tumor. In contrast, a headache is rarely a symptom for cold, but most people manage to catch a cold every single year. Given no other information, do you think it more likely that the headache is caused by a tumor, or by a cold?
If you thought a cold was more likely, well, that was the answer I was after. Even if a brain tumor caused a headache every time, and a cold caused a headache only one per cent of the time (say), having a cold is so much more common that it's going to cause a lot more headaches than brain tumors do. Bayes' theorem, basically, says that if cause A might be the reason for symptom X, then we have to take into account both the probability that A caused X (found, roughly, by multiplying the frequency of A with the chance that A causes X) and the probability that anything else caused X. (For a thorough mathematical treatment of Bayes' theorem, see Eliezer's Intuitive Explanation.)
There should be nothing surprising about that, of course. Suppose you're outside, and you see a person running. They might be running for the sake of exercise, or they might be running because they're in a hurry somewhere, or they might even be running because it's cold and they want to stay warm. To figure out which one is the case, you'll try to consider which of the explanations is true most often, and fits the circumstances best.
Core tenet 1: Any given observation has many different possible causes.
Acknowledging this, however, leads to a somewhat less intuitive realization. For any given observation, how you should interpret it always depends on previous information. Simply seeing that the person was running wasn't enough to tell you that they were in a hurry, or that they were getting some exercise. Or suppose you had to choose between two competing scientific theories about the motion of planets. A theory about the laws of physics governing the motion of planets, devised by Sir Isaac Newton, or a theory simply stating that the Flying Spaghetti Monster pushes the planets forwards with His Noodly Appendage. If these both theories made the same predictions, you'd have to depend on your prior knowledge - your prior, for short - to judge which one was more likely. And even if they didn't make the same predictions, you'd need some prior knowledge that told you which of the predictions were better, or that the predictions matter in the first place (as opposed to, say, theoretical elegance).
Or take the debate we had on 9/11 conspiracy theories. Some people thought that unexplained and otherwise suspicious things in the official account had to mean that it was a government conspiracy. Others considered their prior for "the government is ready to conduct massively risky operations that kill thousands of its own citizens as a publicity stunt", judged that to be overwhelmingly unlikely, and thought it far more probable that something else caused the suspicious things.
Again, this might seem obvious. But there are many well-known instances in which people forget to apply this information. Take supernatural phenomena: yes, if there were spirits or gods influencing our world, some of the things people experience would certainly be the kinds of things that supernatural beings cause. But then there are also countless of mundane explanations, from coincidences to mental disorders to an overactive imagination, that could cause them to perceived. Most of the time, postulating a supernatural explanation shouldn't even occur to you, because the mundane causes already have lots of evidence in their favor and supernatural causes have none.
Core tenet 2: How we interpret any event, and the new information we get from anything, depends on information we already had.
Sub-tenet 1: If you experience something that you think could only be caused by cause A, ask yourself "if this cause didn't exist, would I regardless expect to experience this with equal probability?" If the answer is "yes", then it probably wasn't cause A.
This realization, in turn, leads us to
Core tenet 3: We can use the concept of probability to measure our subjective belief in something. Furthermore, we can apply the mathematical laws regarding probability to choosing between different beliefs. If we want our beliefs to be correct, we must do so.
The fact that anything can be caused by an infinite amount of things explains why Bayesians are so strict about the theories they'll endorse. It isn't enough that a theory explains a phenomenon; if it can explain too many things, it isn't a good theory. Remember that if you'd expect to experience something even when your supposed cause was untrue, then that's no evidence for your cause. Likewise, if a theory can explain anything you see - if the theory allowed any possible event - then nothing you see can be evidence for the theory.
At its heart, Bayesianism isn't anything more complex than this: a mindset that takes three core tenets fully into account. Add a sprinkle of idealism: a perfect Bayesian is someone who processes all information perfectly, and always arrives at the best conclusions that can be drawn from the data. When we talk about Bayesianism, that's the ideal we aim for.
Fully internalized, that mindset does tend to color your thought in its own, peculiar way. Once you realize that all the beliefs you have today are based - in a mechanistic, lawful fashion - on the beliefs you had yesterday, which were based on the beliefs you had last year, which were based on the beliefs you had as a child, which were based on the assumptions about the world that were embedded in your brain while you were growing in your mother's womb... it does make you question your beliefs more. Wonder about whether all of those previous beliefs really corresponded maximally to reality.
And that's basically what this site is for: to help us become good Bayesians.
Continuing.
Part of the great danger in explaining a High topic is that people who haven't been able to understand High topics are super wary about looking like an idiot. Math is the most obvious High topic that people hate trying to understand. They would much rather admit to fearing math than trying and failing at understanding it.
This is sad, to me, because math isn't really that hard to understand. It is a daunting subject that never ends but the fundamentals are already understood by anyone who functions in society. They just never put all the pieces together with the right terms.
I am firmly convinced that the Way of Bayes is like this. The sequences are, for the most part, about subjects that could be easy to understand. They make intuitive sense. The details and the numbers are a pain, but the concept itself is something I could explain to nearly everyone I know. (So I think. I haven't actually tried yet.)
A sentence like the one I quoted above is one that will put a layperson on defensive. This pushes Bayesianism into the realm of High topics: Topics that are grasped by the Smart people; the intellectual elite. Asking them questions at all makes them realize they don't know the answer. This is scary. Immediately answering the question and telling them the answer should be obvious could easily make them feel awkward, even if they got the answer correct.
Articles explaining "obvious" things are often explaining not-obvious things and assume that you are following them each step of the way. These articles are full of trick questions and try to make you second guess yourself in an effort to show you what you do not know. This is scary and elitist to someone who has sold their own intelligence short.
Your example is so minor that most people wouldn't have a problem with it. I bring it up because I am picky. This is an example of aiming far, far too high. The audience at LessWrong reads a question/answer like this and enjoys it. They like learning they are wrong and revel in the introspection that follows as they chase down the error in the machine so they can fix it. A layperson dreads this. They think it means they are stupid and unable to understand. They fail at the competition of intelligence whether the competition actually exists or not.
I think this belongs in the description of the example. You could even leave out the actual numbers because they only matter for the people that have the exact numbers. It takes too long to explain that you just made the numbers up because:
And... the layperson just zoned out. This is the big obstacle in trying to describe Bayesianism. Math scares people away. Even people who are good at math will glaze over when they see As and Xs and words like "probability." I have no idea how to get around this obstacle, honestly. Your attempt was solid. But I still think this is the paragraph where you will lose the lowest rung of your audience.
What if they were surprised? What if their whole world reeled at the question of what causes headaches? What if, horrifically, they completely misunderstood the previous example and are currently pondering if their headache means they have a brain tumor?
If they are completely bewildered right now, telling them they shouldn't be surprised will make them feel dumb. Even if they are dumb, your article shouldn't make them feel dumb. It should make them feel smart.
I don't think this example clarifies much. A bullet list:
More coming if you still want it. My lunch break is over. :)
Very interesting. Actually, I didn't seek to aim that low - I was targeting the average LW reader (or at least an average person who was comfortable with maths). However, I still find this to be very valuable, since I have played around with the idea of trying to write a book that'd attempt to sell (implicitly or explicitly) the idea of "maths / science, especially as applied to rationality / cognitive science is actually fun" to a lay audience.
So I probably won't alter the original article as a reaction to this, but if you want to nevertheless help me in figuring out how to reach to that audience, do continue. :)