For a while I've wanted to explain to people why the testimony of the apostles isn't sufficient for me to believe them. I intuitively find that the gospels aren't enough for be to believe them. Even if they're better preserved than any other ancient text etc. I usually go with extraordinary claims require extraordinary evidence, which is a good heuristic and is convincing to me, but I didn't have a good mathematical explanation of why it's so. I could come up with various reasons of why it's a good idea, but not a *proof*. It turns out that this is just yet another result of Bayes' rule...
Telepathy
Dr Samuel Soal was a parapsychologist who did a number of ESP experiments between 1941-1943 to discover whether it was possible to read someones mind. The basic setup was an agent with a set of 5 animal cards which were shuffled before each test. The agent would pick one at random, and the subject would try to guess what card was picked. Naively, one would expect the subject to be right around once in 5 trials, i.e. p = 0.2. This is not what Soal found. A Mrs Steward, who was the test subject, managed to correctly guess the card 9410 times out of 37100 trials (I'm impressed by her patience), which is a success rate of f = 0.2536. This is 25.8 standard deviations away from the expected success rate, which is a big deal. So we have:
- P(H0|X) - the null hypothesis that she should have random results with a success rate of 0.2 - P(Hf|X) - an alternative hypothesis that she is an ESPer if the success rate is 0.2535
Plugging the values into the appropriate equations (check page 121 of the book), it turns out that the probability of getting this data with the null hypothesis is some 3.15×10−139. So it appears that the only thing left is to accept that ESP is real, given the following equation:
P(Hf|DX)=P(Hf|X)P(D|HfX)P(D|X), where P(D|X)=P(DH0|X)+P(DHf|X)
Since P(D|H0X) is absurdly small, this can be approximated as P(Hf|DX)=P(Hf|X)P(D|HfX)P(Hf|X)P(D|HfX)≈1, as P(H0|DX)≈0. Not very helpful. This pretty much says that if your prior of P(Hf|X) isn't too small, you're going to end up with Objective Scientific Evidence that ESP is real.
Getting real
The obvious problem with the above calculations is that those aren't the only two possible hypotheses. An alternative would be that they cheated. Or that the cards weren't shuffled properly. Or a mistake was made while writing down the answers. Or that the agent subconsciously reacted differently to one of the cards (this one would actually be quite interesting). This changes the basic equation from
P(Hf|DX)=P(Hf|X)P(D|HfX)P(D|X), where P(D|X)=P(DH0|X)+P(DHf|X)
to
P(Hf|DX)=P(Hf|X)P(D|HfX)P(D|X), where P(D|X)=P(DH0|X)+P(DHf|X)+∑na=0P(DHa|X), n is the number of alternatives.
This means that for P(Hf|DX) to have a high value (i.e. close to 1), P(D|X) has to be low, or rather to be massively dominated by P(DHf|X). Which in turn depends on your priors and the relative likelihood of all alternatives. If you assume that there are some 100 plausible alternatives which are all more likely than ESP, then they quickly eclipse the evidence for Hf. Even if a given alternative is shown to be invalid, its probability mass will simply get divided among all the alternatives, which won't change much (unless you can exclude all the possible alternatives, of course). Striking down one alternative will just raise a different one that is more likely on your priors. Not that this is impossible - meteorites were once viewed as superstitious bunk until overwhelming evidence appeared for the contrary.
Wonders and signs
For a while I've wanted to explain to people why the testimony of the apostles isn't sufficient for me to believe them. I intuitively find that the gospels aren't enough for be to believe them. Even if they're better preserved than any other ancient text etc. I usually go with extraordinary claims require extraordinary evidence, which is a good heuristic and is convincing to me, but I didn't have a good mathematical explanation of why it's so. I could come up with various reasons of why it's a good idea, but not a *proof*. It turns out that this is just yet another result of Bayes' rule...
Telepathy
Dr Samuel Soal was a parapsychologist who did a number of ESP experiments between 1941-1943 to discover whether it was possible to read someones mind. The basic setup was an agent with a set of 5 animal cards which were shuffled before each test. The agent would pick one at random, and the subject would try to guess what card was picked. Naively, one would expect the subject to be right around once in 5 trials, i.e. p = 0.2. This is not what Soal found. A Mrs Steward, who was the test subject, managed to correctly guess the card 9410 times out of 37100 trials (I'm impressed by her patience), which is a success rate of f = 0.2536. This is 25.8 standard deviations away from the expected success rate, which is a big deal. So we have:
- P(H0|X) - the null hypothesis that she should have random results with a success rate of 0.2
- P(Hf|X) - an alternative hypothesis that she is an ESPer if the success rate is 0.2535
Plugging the values into the appropriate equations (check page 121 of the book), it turns out that the probability of getting this data with the null hypothesis is some 3.15×10−139. So it appears that the only thing left is to accept that ESP is real, given the following equation:
P(Hf|DX)=P(Hf|X)P(D|HfX)P(D|X), where P(D|X)=P(DH0|X)+P(DHf|X)
Since P(D|H0X) is absurdly small, this can be approximated as P(Hf|DX)=P(Hf|X)P(D|HfX)P(Hf|X)P(D|HfX)≈1, as P(H0|DX)≈0. Not very helpful. This pretty much says that if your prior of P(Hf|X) isn't too small, you're going to end up with Objective Scientific Evidence that ESP is real.
Getting real
The obvious problem with the above calculations is that those aren't the only two possible hypotheses. An alternative would be that they cheated. Or that the cards weren't shuffled properly. Or a mistake was made while writing down the answers. Or that the agent subconsciously reacted differently to one of the cards (this one would actually be quite interesting). This changes the basic equation from
P(Hf|DX)=P(Hf|X)P(D|HfX)P(D|X), where P(D|X)=P(DH0|X)+P(DHf|X)
to
P(Hf|DX)=P(Hf|X)P(D|HfX)P(D|X), where P(D|X)=P(DH0|X)+P(DHf|X)+∑na=0P(DHa|X), n is the number of alternatives.
This means that for P(Hf|DX) to have a high value (i.e. close to 1), P(D|X) has to be low, or rather to be massively dominated by P(DHf|X). Which in turn depends on your priors and the relative likelihood of all alternatives. If you assume that there are some 100 plausible alternatives which are all more likely than ESP, then they quickly eclipse the evidence for Hf. Even if a given alternative is shown to be invalid, its probability mass will simply get divided among all the alternatives, which won't change much (unless you can exclude all the possible alternatives, of course). Striking down one alternative will just raise a different one that is more likely on your priors. Not that this is impossible - meteorites were once viewed as superstitious bunk until overwhelming evidence appeared for the contrary.
Lessons
- Soal cheated
- Read Probability Theory: The Logic of Science
- Lee Strobel isn't convincing since he has different priors
- AI risk and other outlandish claims are unbelievable because people have different priors