Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

LauraABJ comments on Parapsychology: the control group for science - Less Wrong

62 Post author: AllanCrossman 05 December 2009 10:50PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (185)

You are viewing a single comment's thread. Show more comments above.

Comment author: LauraABJ 06 December 2009 12:55:36PM 5 points [-]

Exactly! I guess Allan needs to explain further why parapsychology is bunk. As an example, a person 'reading the mind' of another person a mile away without the emission of any kind of detectable electromagnetic wave or signal capable of traveling that far is in violation of the laws of physics as we know them (and if people did emit such signals, it would be intensively studied). For this to be true, physics itself would need to be complicated on a level that would specifically allow this phenomenon to occur, which seems very, very unlikely. To quote Michael Vassar, "Magic is the hypothesis that physics is complicated."

We should expect positive results from the field of parapsychology, since so many people (in total over the years) are trying to prove it exists, and there is an extreme positive results bias. Thus by chance positive results will be obtained and published, while negative results are largely ignored or not even submitted (I assume a 'scientist' trying to prove parapsychology wants to do so, and so may only bother submitting a paper on the topic if the results are positive).

Comment author: Kaj_Sotala 07 December 2009 09:51:56AM *  3 points [-]

...Charles Honorton and his colleagues drew together all the forced-choice experimental precognition experiments reported in English between 1935 and 1987, publishing their findings in the December 1989 Journal of Parapsychology. The combined results were impressive: 309 studies contributed to by 62 senior authors and their associates, nearly two million individual trials made by more than 30,000 subjects. (In a properly conservative culling, all the experimental work of both Rhine's chosen but subsequently disgraced successor, Walter J. Levy, and S.G. Soal, once a famous specialist in time-displacement psi tests, was excluded; both were known to have cheated in at least some experiments.) Overall, the cumulation is highly significant - 30 percent of studies provided by 40 investigators were independently significant at the 5 percent level. Yet this was not due to a suspicious handful of successful researchers: 23 of the 62 (37 percent) found overall significant scoring.

By the same token, admittedly, this means 63 percent failed to show significant psi. But [...] [i]f one hundred studies are done, averaging as many as thirty-eight correct calls instead of the twenty-five due to chance, then, surprisingly, we should only expect to find among that one hundred "about 33 [statistically] significant studies ... and a 30% chance that there would be 30 or fewer!" Here's why: The scattergun variance that arises simply from chance would mask most of the extra correct calls. This fact would remain in force even if the responders were picking up their extra hits through hidden radio receivers rather than psi! It's just what happens with the statistics of phenomena that have low power. [...]

Well, could this 37 percent success rate be due to the "file drawer"? Hardly. Honorton's estimate required fourty-six unreported chance-level experiments for each of those in the meta-study, including those that themselves gave no significant support for the paranormal hypothesis. It seems highly unlikely that such a trove of dull experiments exists [...] Nor were the results due to an excessive contribution from a few specialist parapsychologists doing so many precognition studies that their non-scoring rivals were swamped. Strikingly, if all the investigators "contributing more than three studies are eliminated, leaving 33 investigators, the combined z [number of standard deviations found] is still 6.00" - with an associated probability of chance coincidence of somewhat more than one in a billion.

The individual effect sizes were all over the place, so Honorton and his coauthor, Diane C. Ferrari, unceremoniously threw out all the studies with unusually large deviations from the mean. [...] "Outcomes remain highly significant. Twenty-five percent of the studies (62/248) show overall significant hitting at the 5% level." Maybe the quality of studies explains the persistance of apparent anomalies? [...] if anything, the significance of the results climbed as quality improved. [...] What's more, the "effect size" had persisted over more than fifty years. This measure compensates for the different sample sizes in various studies: technically, it divides the z score by the square root of the number of trials in each study.

-- Damien Broderick, Outside the Gates of Science

Comment author: Jack 07 December 2009 10:14:25AM *  2 points [-]

Would it really surprise anyone here if, say, 10 percent of parapsychologists are either rigging experiments, hiding negative results or falsifying data? 20%?

Thirty-seven percent.

Comment author: Kaj_Sotala 07 December 2009 04:13:47PM *  4 points [-]

What would be the incentive? Forging results for highly public performances that allowed you to make money off people, sure. But for results published in obscure journals, when even academics in well-respected fields may need to fight tooth and claw for their next yearly funding? In a field that won't even get you the respect of most other academics, and might very well ruin your scientific reputation? Trying to prove a view that doesn't have powerful ideological backers pouring money into it the way creationists do? And with the number of fake researchers apparently staying roughly even for a period of fifty years, looking from the way the effect size hasn't changed?

Comment author: Jack 07 December 2009 05:50:18PM 6 points [-]

And with the number of fake researchers apparently staying roughly even for a period of fifty years, looking from the way the effect size hasn't changed?

That right there is a really good point I didn't think of. As for motive, my impression is that a lot of parapsychologists are trying to demonstrate the truth of beliefs that are incredibly significant to them-- their new age spirituality is at stake. For that matter, if they've dedicated their lives to the subject. If there are no psychic phenomena they have literally spent their lives studying nothing. You might as well ask why theologians never come up with arguments disproving the existence of God. But your point about consistency makes this all moot. I'll check out the book.

Comment author: AllanCrossman 07 December 2009 06:43:12PM 5 points [-]

why theologians never come up with arguments disproving the existence of God

Well if they do they get called philosophers of religion instead...

Comment author: CarlShulman 16 March 2012 12:49:31AM *  2 points [-]

What would be the incentive?

To get more funding for their work, more fame within the parapsychology community, and to make it more likely that the world at large will realize the truth via "fake-but-accurate" experiments. Some parapsychologists pay for their own experiments, using resources garnered from a "day job" in some other field, but many rely on donations from wacky psi-enthusiasts (people who also get excited about ghosts, "subtle energies" and so forth), or selling psi-controlled meditation lamps. Many others think that it's critically important for mainstream funding sources to provide grants to parapsychologists (such as themselves) to do the work they find interesting and important.

Under those circumstances, a psychic believer could come up with all sorts of justifications:

I have to publish these "fake but accurate" experiments to convince others of the effects that I KNOW are really there, and thus gain enough resources to get definitive proof. After all, surely those dishonest skeptics and materialists (who regularly misrepresent the existing literature, and deceive the broader scientific community about the great work done in parapsychology) are doing the same thing, and if only one side 'enhances' its data then the truth will lose out.

Comment author: CarlShulman 16 March 2012 01:24:02AM *  1 point [-]

Honorton's estimate required fourty-six unreported chance-level experiments for each of those in the meta-study, including those that themselves gave no significant support for the paranormal hypothesis.

Note that this is a bogus calculation: it says that if there was no publication bias, so that unpublished studies were just as likely to show positive results as published ones, then adding the stated number of chance studies would "dilute" the results below a threshold significance level. But of course the whole point of publication bias is the enrichment of the file-drawer with negative results. See this paper by Scargle. You need far fewer studies in the file-drawer given the presence of bias. Further, various positive biases will be focused in the published literature, e.g. people doing outright fraud will normally do it for an audience.

The number of studies needed also collapses if various questionable research practices (optional stopping, post hoc reporting of subgroups as separate experiments, etc) are used to concentrate 'hits' into some experiments while misses can be concentrated in a small file drawer.

Parapsychologists counter that the few attempts to audit for unpublished studies (which would not catch everything) have not found large skew in the unpublished studies, but these inflated "fail-safe" statistics are misleadingly large regardless.

Comment author: LauraABJ 07 December 2009 02:24:47PM 1 point [-]

"Honorton's estimate required fourty-six unreported chance-level experiments for each of those in the meta-study, including those that themselves gave no significant support for the paranormal hypothesis."

Why is this at all unlikely? This is a 52 year span of time, and who knows how many times each of these (only 62) 'scientists' ran the trials or tweaked the procedure before they decided they had a set of data worth submitting. Who knows how many people looked for these phenomena, didn't find them, and gave up without submission? Even without outright fraud (which I wouldn't doubt), people lie to themselves. I've worked with scientists who had evidence that their previously obtained results were bunk and submitted them anyway... 'maybe the retest was flawed...' The significant effect that was found may just be the threshold at which an investigator needs to see (or fake) results to submit a paper. There's the answer to the question Allan originally posed...

Also, on another note, not all 'forced choice' tests are conducted in the same way. Some of them involve the person looking at the card being in the same room as the guesser, and well, it's not hard to imagine ways of getting a score above chance like that.

Comment author: Kaj_Sotala 07 December 2009 04:05:01PM *  0 points [-]

Why is this at all unlikely? This is a 52 year span of time

309 times 46 is 14 214, which divided by 52 equals approximately 273 unpublished studies per year. I haven't seen any figures on how many studies were conducted for e.g. a specific experimental paradigm in psychology during that time, so I can't say for certain how plausible this is or isn't. It does sound a bit high considering that parapsychology hasn't exactly been the best-funded field around, though it might have had more money in the 1930's. Does anyone have numbers?

Comment author: LauraABJ 07 December 2009 07:13:53PM 2 points [-]

I'm not saying there were 14k unpublished completed full studies, I'm suggesting that what got published was already biased. There is room for selection bias at every level of a study, including which trials and which methods are finally taken, written up, and submitted. If the 'scientists' are trying to prove that psi exists, they can find it, one or another. Fraud isn't even required, just wishful thinking. The consistency of the effect is interesting, but may only be measuring the psychological phenomenon of deliberate self-deception.-- ah we've discovered the threshold deviation from chance at which people will believe their own crap hasn't been tampered by their own meddling.

Think about the alternative explanation: If the forced choice test is run properly- the subject guesses which order 5 symbols in a 25 card deck will appear before the deck has been shuffled. The deck is then shuffled by machine (or associate) in a different room, and the order of the cards are examined. Now, how do you propose the subject is entangled with the card shuffling machine and deck, without violating current physical law, such that he can predict the order? This is magical thinking, with no basis in reality as we know it. Unless there is a pattern in the card shuffling machine and some people are very aware of it due to practice with it... but that is hardly psi.

Comment author: Kaj_Sotala 07 December 2009 08:39:28PM *  2 points [-]

Think about the alternative explanation

I'm not saying that psi must be real, only that it seems to merit a closer look than most people in this thread have been implying. Yes, it does seem rather unlikely that psi would exist, which is why I'm still undecided myself. But the fact that we can't come up with any physical explanation for it doesn't mean that it couldn't be real. As Yvain pointed out, Newton's theories may at the time have seemed like magical thinking as well. There could be some physical mechanism we're just not aware of, but which the brain has nonetheless evolved to take advantage of.

Or then it might just be a sign of our statistical methods being flawed, made worse by psi researchers being insufficiently rigorous in their methods.

Comment author: LauraABJ 08 December 2009 01:30:23AM 6 points [-]

"I'm not saying that psi must be real, only that it seems to merit a closer look than most people in this thread have been implying."

I strongly disagree. Psi has been looked at very intensively for a very long time, and the best it can yield is that it's not completely statistically insignificant. No theories have been posed as to how it works, it hasn't been quantified (ie, how far away, in what time frame can the subject predict the future), and it cannot be demonstrated reliably and repeatedly from even a few individuals who could then be studied more elaborately. Even one person who could always predict the order of a deck of cards would be fascinating. At some point, you just have to say a line of research does not merit further study.

In the mean time, giving these theories credence wastes time and resources and leads people to think they can believe anything they want about the world, including the outstanding religious dogma, since, hey, you never know.

Comment author: scav 09 December 2009 01:30:52PM 0 points [-]

Here's a closer look: to accept psi, you would have to reject evolutionary biology.

It would be such a humungous advantage to communicate telepathically, see the future or remote locations, or manipulate the physical world by thinking, that there's no way evolution wouldn't have optimised the f* out of it by now.

We don't wonder whether birds have wings, or whether dogs have a sense of smell. That we can wonder whether we might have psychic powers means we DON'T, to a very high probability indeed.

Comment author: Kaj_Sotala 09 December 2009 04:00:52PM *  6 points [-]

Just because an ability would be useful doesn't mean that evolution could (or would, if reaching that ability required several intermediate steps with very low fitness advantages) optimize it without limit.

The ability to digest literally everything we put in our mouths would be useful as well, but the fact that we don't have that doesn't mean we need to reject evolution.

Comment author: scav 11 December 2009 04:07:06PM 0 points [-]

Voted up for making me think harder.

I'm not talking about an ability (like digesting cellulose) which would be really advantageous but we don't have and would require a lot of unlikely steps. The non-null hypothesis of human psychic powers is that we do already have them and ancient humans did too. Yet we don't seem to have evolved psychic abilities that are even detectable by now.

Compare: the abilies to cope with milk and beer in our diet have been evolving in humans since the invention of dairy farming and brewing (a few thousand years ago?) There is large population variation in these digestive abilities after that short time.

Would the selection pressure in favour of telepathy be that much less than for drinking beer?

Comment author: Blueberry 13 December 2009 07:38:06AM 1 point [-]

The problem here is that you're assuming a) psychic abilities would have some degree of heritability, instead of being random accidents that aren't passed on genetically, and b) that psychic abilities can vary in degree, so that there could be selection pressure to make them larger, instead of being binary.

Also consider that psychic abilities in small amounts could have detrimental effects on fitness: for instance, they could make you more sensitive to bad moods, more temperamental, or even insane.

Comment author: Strange7 05 February 2014 05:42:47AM 0 points [-]

Beer consumption has all sorts of implications for social interaction and waterborne disease, and in some environments, there are no close substitutes. Digestive efficiency is a major factor in survival, one way or another; not being able to cope with the food and drink you've got can kill you, and synthesizing a lot of tricky enzymes you don't need (or, equivalently, hosting intestinal flora which aren't pulling their weight) can also kill you.

Telepathy, on the other hand, doesn't seem to involve enough of the body to have significant metabolic effects one way or another, and is unreliable and vague even for the best performers. What life-or-death/reproduce-or-don't outcomes would it be exerting selection pressure through?

Comment author: timtyler 09 December 2009 04:15:08PM 5 points [-]

Cellphones are advantageous too - yet their evolution seems to be currently in progress. That should be enough to indicate why this line of argument fails.

Comment author: Vladimir_Nesov 08 December 2009 10:33:13AM *  0 points [-]

I'm not saying that psi must be real, only that it seems to merit a closer look than most people in this thread have been implying.

This does mean estimating it to be much more probably real than seems reasonable at this point.