New comments on the recent psi study

14 [deleted] 23 November 2010 03:52PM

HT reddit/r/science: http://www.ruudwetzels.com//articles/Wagenmakersetal_subm.pdf

Probably nobody is surprised here, but I thought one might be interested.

Comments (5)

Comment author: Risto_Saarelma 23 November 2010 04:05:59PM 4 points [-]
Comment author: jimrandomh 23 November 2010 05:16:49PM 9 points [-]

That response boils down to "abusing statistics is okay because other fields are doing it too". But the fact that other scientific fields are also abusing statistics does not make it okay, because it does not make the conclusions that result from statistical abuses true. The choice of which statistical test to use is not arbitrary, and using the wrong one is as bad as writing down the wrong value for a low-order digit; you can get away with it when the effect size is large, but not here.

(Paraphrased from my reply on that article's comments section)

Comment author: RobinZ 24 November 2010 04:38:52AM 6 points [-]

From the blog post:

Using a different sort of statistical test than Bem used, they re-analyze Bem's data and they find that, while the results are positive, they are not positive enough to pass the level of "statistical significance." They conclude that a somewhat larger sample size would be needed to conclude statistical significance using the test they used.

Err, that's not what they found. Over half the data was not merely "not positive enough", but literally negative.

Comment author: DSimon 23 November 2010 04:19:28PM 4 points [-]

That article is actually a really good introduction to the advantages of Bayesian statistics in experiments over the regular p-value approach.

Comment author: DanielVarga 23 November 2010 08:15:47PM 7 points [-]

And it is an even better introduction to proper experiment design. In particular, it argues eloquently for a very clear distinction between exploratory and confirmatory experiments. This distinction should be drilled into every junior undergrad, but somehow even world-famous experimental psychologists can miss it. Now, if we asked Daryl Bem about this, he would probably say that all his experiments were exploratory, and now it is the task of the research community to confirm or reject them. But the problem is, he used significance tests that were only suitable for confirmatory experiments.