The statistical error that just keeps on coming, Ben Goldacre, Guardian, Friday 9 September 2011 20.59 BST
We all like to laugh at quacks when they misuse basic statistics. But what if academics, en masse, deploy errors that are equally foolish? This week Sander Nieuwenhuis and colleagues publish a mighty torpedo in the journal Nature Neuroscience.
They've identified one direct, stark statistical error so widespread it appears in about half of all the published papers surveyed from the academic psychology research literature.
[...]
How often? Nieuwenhuis looked at 513 papers published in five prestigious neuroscience journals over two years. In half the 157 studies where this error could have been made, it was. They broadened their search to 120 cellular and molecular articles in Nature Neuroscience, during 2009 and 2010: they found 25 studies committing this fallacy, and not one single paper analysed differences in effect sizes correctly.These errors are appearing throughout the most prestigious journals for the field of neuroscience.
Update: Erroneous analyses of interactions in neuroscience: a problem of significance (PDF)
I was trying to say more than we actually could say. I was the one analyzing the data.
We were studying the growth of neurons and patterns of gene regulation under the presence of various inhibitors of the mechanism of depolarization regulation. Say we had inhibitors A and B.
We would have done expression profiles for cultures that were, for example, this:
5 mM KCl (physiological conditions); 25 mM KCl (depolarizing conditions); 25 mM KCl + inhibitor A; 25 mM KCl + inhibitor B; 25 mM KCl + inhibitor A and B;
(Upon editing, I can't make those appear as a line by line list for some reason? Don't know what's up with that.)
I wanted to make comparisons between the A+B+25 cultures and the 25 mM Culture (and the 5 mM one), but you can't do that. You can only compare A+B to it's controls, A and B alone.