Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored - that they are unfairly being held to higher standards than everyone else. I'm willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.
— Eliezer Yudkowsky, Frequentist Statistics are Frequently Subjective
Imagine if, way back at the start of the scientific enterprise, someone had said, "What we really need is a control group for science - people who will behave exactly like scientists, doing experiments, publishing journals, and so on, but whose field of study is completely empty: one in which the null hypothesis is always true.
"That way, we'll be able to gauge the effect of publication bias, experimental error, misuse of statistics, data fraud, and so on, which will help us understand how serious such problems are in the real scientific literature."
Isn't that a great idea?
By an accident of historical chance, we actually have exactly such a control group, namely parapsychologists: people who study extra-sensory perception, telepathy, precognition, and so on.
There's no particular reason to think parapsychologists are doing anything other than what scientists would do; their experiments are similar to those of scientists, they use statistics in similar ways, and there's no reason to think they falsify data any more than any other group. Yet despite the fact that their null hypotheses are always true, parapsychologists get positive results.
This is disturbing, and must lead us to wonder how many positive results in real science are actually wrong.
The point of all this is not to mock parapsychology for the sake of it, but rather to emphasise that parapsychology is useful as a control group for science. Scientists should aim to improve their procedures to the point where, if the control group used these same procedures, they would get an acceptably low level of positive results. That this is not yet the case indicates the need for more stringent scientific procedures.
Acknowledgements
The idea for this mini-essay and many of its actual points were suggested by (or stolen from) Eliezer Yudkowsky's Frequentist Statistics are Frequently Subjective, though the idea might have originated with Michael Vassar.
This was originally published at a different location on the web, but was moved here for bandwidth reasons at Eliezer's suggestion.
Comments / criticisms
A discussion on Hacker News contained one very astute criticism: that some things which may once have been considered part of parapsychology actually turned out to be real, though with perfectly sensible, physical causes. Still, I think this is unlikely for the more exotic subjects like telepathy, precognition, et cetera.
You're right that I completely missed the Bayesian boat, and I'm going to have to start thinking more before I speak and revise my estimates down to <1%.
But I'm still reluctant to put them as low as you seem to. The anthropic principle combined with large universe says that whatever complexity is necessary for the existence of conscious observers, we can expect to find at least that level of complexity. Questions like consciousness, qualia, and personal identity still haven't been resolved, and although past experience suggests there is probably a rational explanation to this question, it isn't nearly dissolved yet. If consciousness really is impossible without some exotic consciousness-related physics (Penrosean or otherwise), then our universe will have exotic consciousness-related physics no matter how complex they need to be. And since evolved beings have been so proficient at making use of normal physics to gain sensory information, it's a good bet they'd do the same with exotic consciousness-related physics too if they had them...
...is a somewhat hokey argument I just invented on the spot, and I'm sorry for it. But the ease with which I can put something like that together is itself evidence that there are enough possible sides of the issue that hadn't been considered (at least I hadn't considered that one; maybe you've been thinking about it for years) that it needs at least a little more room for error than two in a bajillion (sorry, Alex).
I also disagree with your assessment of the amount of evidence. Have you ever read any good books by intelligent believers in the subject? It's not all John Edwards psychic chat shows. I also think you might be double-counting evidence against psi here - psi doesn't exist so we know any apparent evidence must come from human psychology, therefore there never was any apparent evidence in the first place. Or have you read the studies and developed separate explanations for each positive result?
Anyway, let's settle this the LW way. Give me your odds that psi exists, and we can make a bet at them. If it's one in a million, then I'll give a cent to your favorite charity on the condition that you give $10,000 to my favorite charity if psi's shown to exist within our lifetimes (defined however you want; possibly as evidence sufficient to convince any two among Randi, Dawkins, and Eliezer that psi is >50% likely).
Psi doesn't even explain consciousness or qualia.
[edit] Oops, necro. Disregard me.
[edit edit] okay! nevermind that then :D