Parapsychologists make a poor control group of scientists because part of their job is collecting evidence that parapsychology works. In science, that step is already done. Biologists do not need to prove that life works, because life exists. Physicists do not need to prove that physics works, because physics, by definition, IS the way the universe works. Einstein did not dream up relativity and then start looking for evidence to support it. He looked at the evidence that was available, and came up with relativity as a way to explain it. Parapsychologists do it the other way around.
Yes, I know that's what you meant, but the point of my comment is that unlike with parapsychology, the object of study for climatologists definitely exists, whether or not AGW is real. Any individual climatologist who started to think that AGW might not be real won't think "this is the end for climatology", because climate exists either way.
And of course if they worry that funding for climate research might drop as a result of what they've found, they can console themselves with the knowledge that their new belief allows them to earn twice as much money working for an oil-industry funded body such as the CEI.
In no way do I think that the parapsychologists have good hypotheses or reasonable claims. I also am a firm adherent to the ethos: Extraordinary claims must have extraordinary proofs. However to state the following:
one in which the null hypothesis is always true.
is making a bold statement about your level of knowledge. You are going so far as to say that there is no possible way that there are hypotheses which have yet to be described which could be understood through the methodology of this particular subgroup. This exercise seems to me to be rejecting these studies intuitively,(without study) just from the ad hominem approach to rejection - well they are parapsychologists therefore they are wrong. If they are wrong, then proper analysis would indicate that, would it not?
I have never seen a parapsychology study, so I will go look for one. However does every single study have massive flaws in it?
I have never seen a parapsychology study, so I will go look for one. However does every single study have massive flaws in it?
Damien Broderick's Outside the Gates of Science summarizes a number of parapsychology studies, noting that several of the studies do indeed seem quite solid. It doesn't come to any definite conclusion over whether psi phenomena are actually real or if there's just something wrong with our statistical techniques, but it does seem like there might be enough to warrant more detailed study. See also e.g. Ben Goertzel's review of the book.
There are many other things that people have claimed can be rejected intuitively without study through the years.
In the 18th century, everyone knew that real scientific physics only permitted a body to act upon another body through direct contact. When Newton proposed his theory of gravity, many people rejected it as pseudoscientific or magical because it claimed the stars and planets could exert action at a distance, without saying how they did it.
In the 19th century, everyone knew that life was on a different order than mere matter, because obviously you couldn't produce the self-moving and self-regenerating qualities of life with just stuff like you get in rocks and sand.
In the 20th century, everyone knew that the mind was more than just the brain, since simple introspection could determine the existence of a consciousness inexplicable in simple material terms.
The absurdity heuristic is an okay heuristic, but I'd be really really careful before saying something is so absurd we can throw away any contradictory experimental evidence without a glance.
The possibility I give to some sort of psi effect existing (in a nice, scientific way that we can study once we figure out what form...
I used to think that way before I knew about Bayesianism. Once I learned about it I realized that the prior probability for psi was very VERY low, e.g. its complex and there's no reason to expect it so one in a bajillion, while the probability for the observed evidence for psi, given what we know about psychology, was well in excess of 50% in the absence of psi, so the update couldn't justify odds greater than two in a bajillion.
One in a bajilion? Guys, the numbers matter. 10^-9 is very different from 10^-12, which is very different from 10^-15. If we start talking about some arbitrarily low number like "one-in-a-bajillion" against which no amount of evidence could change our mind, then we're really just saying "zero" but not admitting to ourselves that we're doing so.
Other than that, I agree with Yvain and have found this to be perhaps the most belief-changing so far on LW!
It takes only 332 pieces of evidence with likelihood ratios of 2:1 to promote to 1:1 odds a hypothesis with prior odds of 1:googol, that is 10^-100, which would be the appropriate prior odds of something you could describe with around 70 symbols from a 26-letter equiprobable alphabet.
"A bajillion to one" are odds that Bayesian updating can overcome surprisingly quickly - it isn't anything remotely like "no amount of evidence can change my mind". Now odds of one to a googolplex - that might as well be zero, relative to the amount of evidence you could acquire over a human lifetime. But the prior probability of any possibility you can describe over a human lifetime should be much higher than that.
A nitpick: it takes 332 pieces of all mutually independent evidence to perform that level of update.
More confusing, for these purposes the independence level of the evidence depends on what hypotheses you're trying to distinguish with it. E.g. if you're trying to distinguish between "that subject has ESP powers" and "that experiment was random luck" then 332 repetitions of the same experiment will do. If you're trying to distinguish between "that subject has ESP powers" and "that experimenter's facial expressions differ based on what cards he was looking at", then you can't just repeat it; you've got to devise new and different experiments.
You're right that I completely missed the Bayesian boat, and I'm going to have to start thinking more before I speak and revise my estimates down to <1%.
But I'm still reluctant to put them as low as you seem to. The anthropic principle combined with large universe says that whatever complexity is necessary for the existence of conscious observers, we can expect to find at least that level of complexity. Questions like consciousness, qualia, and personal identity still haven't been resolved, and although past experience suggests there is probably a rational explanation to this question, it isn't nearly dissolved yet. If consciousness really is impossible without some exotic consciousness-related physics (Penrosean or otherwise), then our universe will have exotic consciousness-related physics no matter how complex they need to be. And since evolved beings have been so proficient at making use of normal physics to gain sensory information, it's a good bet they'd do the same with exotic consciousness-related physics too if they had them...
...is a somewhat hokey argument I just invented on the spot, and I'm sorry for it. But the ease with which I can put something like that together...
One problem with this argument is that if psi exists, we are very bad at using it, and we don't see other organisms using it well either. The world we see appears to be almost completely described by normal physics at worst.
I don't think that I'm double-counting evidence. I certainly know that there can be intelligent believers, after all, MANY intelligent people believe that one is compelled to accept the conclusions of the scientific method over those of the scientific community. Also, beliefs can be compelling for any variety of irrational reasons. The evidence I have seen though looks to me like exactly the evidence you would expect given known psychology and no psi. We can surely agree that there is a LOT of evidence that hyman psychology would create belief in psi in the absence of psi, can't we.
I would set my odds at "top twenty most astoundingly surprising things ever discovered but maybe not top ten". That seems to me like odds of many billions to one against, but not trillions. Unfortunately, the odds for almost any plausible winning conditions occurring without psi being real are much higher, making the bet difficult to judge. I have a standing 10,0...
Have donated $10 to SIAI (seemed less likely to lose you guys money in transaction fees than $1) with public comment about the bet . Will decide where you can donate your $33000 in the unlikely event it proves necessary.
Well, then I lose the bet...unless someone contacts their ghosts...in which case I win the bet!
In the 20th century, everyone knew that the mind was more than just the brain, since simple introspection could determine the existence of a consciousness inexplicable in simple material terms.
No, they didn't. Superficial research indicates that serious materialism goes back to at least the Enlightenment in the 18th century. And the 20th century? That's not even plausible.
Parapsychology is one of the very few things we can reject intuitively, because we understand the world well enough to know that psychic powers just can't exist.
Do you think it is possible that we are "living in the Matrix"? If so, then you should consider something functionally indistinguishable from psychic powers to be possible.
Real world is uncaring, unsupervised. Magic is not just about the world being "complex", it's about the world containing mechanisms targeting specifically humans, and understanding the situation much like a human would. Being "deterministic" doesn't preclude anything, it's more of a way of seeing things than the way things are.
There's no particular reason to think parapsychologists are doing anything other than what scientists would do; their experiments are similar to those of scientists, they use statistics in similar ways, and there's no reason to think they falsify data any more than any other group. Yet despite the fact that their null hypotheses are always true, parapsychologists get positive results.
This seems self-contradictory. If they get positive results by methods we approve (as you write, nothing other that what scientists do), what entitles us to dismiss those r...
The idea is good. I'm afraid that it may be interpreted as meaning that we need to increase our publication standards from 95% confidence intervals to 98% confidence intervals. I think scientists already have a dangerously strong bias to reject anything that fails to meet a 95% confidence interval. If someone has a good idea, with good theoretical reasoning behind it; and they run some experiments but don't hit 95%, it's still worth considering.
There are also all sorts of data-collection tasks which are routinely thrown out if they fall below 95% confid...
I would like to see more discussions on LW that, like this one, concern the rational conclusion that can be drawn from a particular body of research (as opposed to discussion of rationality itself). I will endeavour to start some such discussions.
Re: "There's no particular reason to think parapsychologists are doing anything other than what scientists would do"
Sure there is. They are studying something that doesn't exist - so they are probably stupider than most scientists, and more likely to believe nonsense, have a relatively poor history of undestanding experimental results - and so on.
A discussion on Hacker News contained one very astute criticism: that some things which may once have been considered part of parapsychology actually turned out to be real, though with perfectly sensible, physical causes.
What gwern said.](http://news.ycombinator.com/item?id=978927
If we assume that approximately p% of results are false positives, and that only positive results are published, then the question becomes how many scientists are trying to prove (and disprove) the same hypothesis. If 1000 scientists are trying to prove that Drug Y slows the progression of Alzheimer's disease, and a p of 0.01 is required for publication, then we need to see more than 10 independent publications supporting this result before we should believe it. Things would be so much easier if negative results were given as much weight as positive ones... Can anyone think of a good way of calibrating the publication bias towards positives?
We could make more productive use of this control if we could compare some other methods as applied to both it and some more standard topic. So far we see that standard academic and "scientific" institutions find results even when there are none. Are there some other institutions that might do better?
Just out of curiosity, has anybody read into the evidence for parapsychology? The best book I've found is "The Conscious Universe" by Dean Radin. There is a critical review of the book here: http://www.skepticreport.com/sr/?p=537
Since when is pre-judging the validity or importance of a subject--in the spirit of "it's obviously nonsense so why even bother studying it?"--considered a "scientific" stance to take? It's dogmatic comments like these that sadly lead many non-scientists to have a less than favorable view on the seeming "objectivity" of the field and its researchers.
The big problem with parapsychology as a field is that science is all of a piece. Thus, physics is consistent with chemistry, biology and so on. So the question is not "what knowledge can we derive on the assumption that we know nothing?" - but "what knowledge can we derive given what we know already?" And we know really quite a lot about areas that directly impinge on this question.
Basic physics leaves it not looking good for parapsychology as a field in any way. Sean M. Carroll points out that both human brains and the spoons they try to bend are made, like all normal matter, of quarks and electrons; everything else they do is properties of the behaviour of quarks and electrons. And normal matter, made of quarks and electrons, interacts through the four forces: strong, weak, electromagnetic and gravitational. Thus either it's one of the four known forces or it's a new force, and any new force with range over 1 millimetre must be at most a billionth the strength of gravity or it will have been captured in experiments already done. So either it's electromagnetism, gravity or something weaker than gravity.
This leaves no force that could possibly account for telek...
Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored - that they are unfairly being held to higher standards than everyone else. I'm willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.
— Eliezer Yudkowsky, Frequentist Statistics are Frequently Subjective
Imagine if, way back at the start of the scientific enterprise, someone had said, "What we really need is a control group for science - people who will behave exactly like scientists, doing experiments, publishing journals, and so on, but whose field of study is completely empty: one in which the null hypothesis is always true.
"That way, we'll be able to gauge the effect of publication bias, experimental error, misuse of statistics, data fraud, and so on, which will help us understand how serious such problems are in the real scientific literature."
Isn't that a great idea?
By an accident of historical chance, we actually have exactly such a control group, namely parapsychologists: people who study extra-sensory perception, telepathy, precognition, and so on.
There's no particular reason to think parapsychologists are doing anything other than what scientists would do; their experiments are similar to those of scientists, they use statistics in similar ways, and there's no reason to think they falsify data any more than any other group. Yet despite the fact that their null hypotheses are always true, parapsychologists get positive results.
This is disturbing, and must lead us to wonder how many positive results in real science are actually wrong.
The point of all this is not to mock parapsychology for the sake of it, but rather to emphasise that parapsychology is useful as a control group for science. Scientists should aim to improve their procedures to the point where, if the control group used these same procedures, they would get an acceptably low level of positive results. That this is not yet the case indicates the need for more stringent scientific procedures.
The idea for this mini-essay and many of its actual points were suggested by (or stolen from) Eliezer Yudkowsky's Frequentist Statistics are Frequently Subjective, though the idea might have originated with Michael Vassar.
This was originally published at a different location on the web, but was moved here for bandwidth reasons at Eliezer's suggestion.
A discussion on Hacker News contained one very astute criticism: that some things which may once have been considered part of parapsychology actually turned out to be real, though with perfectly sensible, physical causes. Still, I think this is unlikely for the more exotic subjects like telepathy, precognition, et cetera.