I've always been really confused by this but it isn't clear that an event with P=0 is an impossible event unless we're talking about the probability of an event in a finite set of possible events. (Edit again: You can skip the rest of this paragraph and the next if you are smarter than me and already get continuous probability distributions. I'm obviously behind today.)This is how it was explained to me: Think of a dart board with a geometric line across it. That line represents probability space. An event with P=.5 is modeled by marking the middle of the line. If someone throws a dart at the line there is an equal chance that it lands at any point along the line. However, at any given point the probability that the dart lands there is zero.
I think the probability of any particular complex entity, event or law existing can be said to have a probability of zero absent a creator or natural selection or some other mechanism for enabling complexity. Of course this is really counterintuitive since our evolved understanding of probability deals with finite sets of possibilities. Also it means that 'impossible' can't be assigned a probability. (Edit: Also, the converse is true. The probability that the dart lands anywhere other than the spot you pick is 1 so certainty can't be mapped as 1 either.)
Also, imperfect Bayesians will sometimes assign less than ideal probabilities to things. A perfect Bayesian would presumably never wrongly declare something impossible because it could envision possible future evidence that would render the thing possible. But regular people are going to misinterpret evidence and fail to generate hypotheses so they might sometimes think something is impossible only to later have it's possibility thrown in their faces.
...I've always been really confused by this but it isn't clear that an event with P=0 is an impossible event unless we're talking about the probability of an event in a finite set of possible events. This is how it was explained to me: Think of a dart board with a geometric line across it. That line represents probability space. An event with P=.5 is modeled by marking the middle of the line. If someone throws a dart at the line there is an equal chance that it lands at any point along the line. However, at any given point the probability that the dart lands
Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored - that they are unfairly being held to higher standards than everyone else. I'm willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.
— Eliezer Yudkowsky, Frequentist Statistics are Frequently Subjective
Imagine if, way back at the start of the scientific enterprise, someone had said, "What we really need is a control group for science - people who will behave exactly like scientists, doing experiments, publishing journals, and so on, but whose field of study is completely empty: one in which the null hypothesis is always true.
"That way, we'll be able to gauge the effect of publication bias, experimental error, misuse of statistics, data fraud, and so on, which will help us understand how serious such problems are in the real scientific literature."
Isn't that a great idea?
By an accident of historical chance, we actually have exactly such a control group, namely parapsychologists: people who study extra-sensory perception, telepathy, precognition, and so on.
There's no particular reason to think parapsychologists are doing anything other than what scientists would do; their experiments are similar to those of scientists, they use statistics in similar ways, and there's no reason to think they falsify data any more than any other group. Yet despite the fact that their null hypotheses are always true, parapsychologists get positive results.
This is disturbing, and must lead us to wonder how many positive results in real science are actually wrong.
The point of all this is not to mock parapsychology for the sake of it, but rather to emphasise that parapsychology is useful as a control group for science. Scientists should aim to improve their procedures to the point where, if the control group used these same procedures, they would get an acceptably low level of positive results. That this is not yet the case indicates the need for more stringent scientific procedures.
Acknowledgements
The idea for this mini-essay and many of its actual points were suggested by (or stolen from) Eliezer Yudkowsky's Frequentist Statistics are Frequently Subjective, though the idea might have originated with Michael Vassar.
This was originally published at a different location on the web, but was moved here for bandwidth reasons at Eliezer's suggestion.
Comments / criticisms
A discussion on Hacker News contained one very astute criticism: that some things which may once have been considered part of parapsychology actually turned out to be real, though with perfectly sensible, physical causes. Still, I think this is unlikely for the more exotic subjects like telepathy, precognition, et cetera.