The Decline Effect and the Scientific Method (article @ the New Yorker)
First, as a physicist, I do have to point out that this article concerns mainly softer sciences, e.g. psychology, medicine, etc.
A summary of explanations for this effect:
- "The most likely explanation for the decline is an obvious one: regression to the mean. As the experiment is repeated, that is, an early statistical fluke gets cancelled out."
- "Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found."
- "Richard Palmer... suspects that an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. ... Palmer emphasizes that selective reporting is not the same as scientific fraud. Rather, the problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results."
- "According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. ... The current “obsession” with replicability distracts from the real problem, which is faulty design."
These problems are with the proper usage of the scientific method, not the principle of the method itself. Certainly, it's important to address them. I think the reason they appear so often in the softer sciences is that biological entities are enormously complex, and so higher-level ideas that make large generalizations are more susceptible to random error and statistical anomalies, as well as personal bias, conscious and unconscious.
For those who haven't read it, take a look at Richard Feynman on cargo cult science if you want a good lecture on experimental design.
I noticed something interesting: in Google Scholar, when you punch in Young as author and the reasonable search terms 'rat' 'maze' 'sand' restricted to before Feynman's lecture, only 3 items pop up.
I don't have access to the 3, so I've requested them: http://lesswrong.com/lw/ji3/lesswrong_help_desk_free_paper_downloads_and_more/auye
(Frustratingly, Young wrote a whole textbook on rats/mice available on the Internet Archive - the year before Feynman says he did the experiment! Another textbook, Emotion in man and animal: its nature and dynamic basis, isn't on IA but is in Google Books; checking it with a few keywords like 'sand' and 'smell' and 'third', doesn't seem to throw up any particularly good hits.)
Emotion in man and animal can now be read on Hathitrust: https://catalog.hathitrust.org/Record/000426365 Checking the ToC doesn't turn up anything relevant, and an additional search for 'maze' shows some maze-running mice experiments but not the one in question.
I wonder if it's possible this is the wrong Young? It is not that rare a US surname (far from it, #28 in 1990). Thinking about it, isn't calling him "Mr. Young" a little odd? P.T. Young definitely had a PhD and was a tenured professor, so it's a bit disrespectful to not refer to him as 'Dr... (read more)