...Charles Honorton and his colleagues drew together all the forced-choice experimental precognition experiments reported in English between 1935 and 1987, publishing their findings in the December 1989 Journal of Parapsychology. The combined results were impressive: 309 studies contributed to by 62 senior authors and their associates, nearly two million individual trials made by more than 30,000 subjects. (In a properly conservative culling, all the experimental work of both Rhine's chosen but subsequently disgraced successor, Walter J. Levy, and S.G. Soal, once a famous specialist in time-displacement psi tests, was excluded; both were known to have cheated in at least some experiments.) Overall, the cumulation is highly significant - 30 percent of studies provided by 40 investigators were independently significant at the 5 percent level. Yet this was not due to a suspicious handful of successful researchers: 23 of the 62 (37 percent) found overall significant scoring.

By the same token, admittedly, this means 63 percent failed to show significant psi. But [...] [i]f one hundred studies are done, averaging as many as thirty-eight correct calls instead of the twenty-five due to chance, then, surprisingly, we should only expect to find among that one hundred "about 33 [statistically] significant studies ... and a 30% chance that there would be 30 or fewer!" Here's why: The scattergun variance that arises simply from chance would mask most of the extra correct calls. This fact would remain in force even if the responders were picking up their extra hits through hidden radio receivers rather than psi! It's just what happens with the statistics of phenomena that have low power. [...]

Well, could this 37 percent success rate be due to the "file drawer"? Hardly. Honorton's estimate required fourty-six unreported chance-level experiments for each of those in the meta-study, including those that themselves gave no significant support for the paranormal hypothesis. It seems highly unlikely that such a trove of dull experiments exists [...] Nor were the results due to an excessive contribution from a few specialist parapsychologists doing so many precognition studies that their non-scoring rivals were swamped. Strikingly, if all the investigators "contributing more than three studies are eliminated, leaving 33 investigators, the combined z [number of standard deviations found] is still 6.00" - with an associated probability of chance coincidence of somewhat more than one in a billion.

The individual effect sizes were all over the place, so Honorton and his coauthor, Diane C. Ferrari, unceremoniously threw out all the studies with unusually large deviations from the mean. [...] "Outcomes remain highly significant. Twenty-five percent of the studies (62/248) show overall significant hitting at the 5% level." Maybe the quality of studies explains the persistance of apparent anomalies? [...] if anything, the significance of the results climbed as quality improved. [...] What's more, the "effect size" had persisted over more than fifty years. This measure compensates for the different sample sizes in various studies: technically, it divides the z score by the square root of the number of trials in each study.

-- Damien Broderick, Outside the Gates of Science