The Decline Effect and the Scientific Method [link]

Dreaded_Anomaly

The Decline Effect and the Scientific Method (article @ the New Yorker)

First, as a physicist, I do have to point out that this article concerns mainly softer sciences, e.g. psychology, medicine, etc.

A summary of explanations for this effect:

"The most likely explanation for the decline is an obvious one: regression to the mean. As the experiment is repeated, that is, an early statistical fluke gets cancelled out."
"Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found."
"Richard Palmer... suspects that an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. ... Palmer emphasizes that selective reporting is not the same as scientific fraud. Rather, the problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results."
"According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. ... The current “obsession” with replicability distracts from the real problem, which is faulty design."

These problems are with the proper usage of the scientific method, not the principle of the method itself. Certainly, it's important to address them. I think the reason they appear so often in the softer sciences is that biological entities are enormously complex, and so higher-level ideas that make large generalizations are more susceptible to random error and statistical anomalies, as well as personal bias, conscious and unconscious.

For those who haven't read it, take a look at Richard Feynman on cargo cult science if you want a good lecture on experimental design.

The Decline Effect and the Scientific Method (article @ the New Yorker)

First, as a physicist, I do have to point out that this article concerns mainly softer sciences, e.g. psychology, medicine, etc.

A summary of explanations for this effect:

"The most likely explanation for the decline is an obvious one: regression to the mean. As the experiment is repeated, that is, an early statistical fluke gets cancelled out."
"Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found."
"Richard Palmer... suspects that an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. ... Palmer emphasizes that selective reporting is not the same as scientific fraud. Rather, the problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results."
"According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. ... The current “obsession” with replicability distracts from the real problem, which is faulty design."

The study mentioned in your first link is most likely "Preferences and demands of the white rat for food", by Paul Thomas Young. This paper is includes a section tantalisingly named "Spatial factors in the feeding behavior of rats", which turns out not to be related to Feynman's story:

Eight rats showed marked individual differences in spatial behavior between the extremes of right and left dominance. The tendency of an animal to eat the test-food in a given position, right or left, frequently appears instead of preferential discrimination. Two attempts were made to control the factor of position.

The path of approach to the test-foods was moved bit by bit to the right or left.

The angular position of the pair of food-containers in relation to the line of approach was varied so as to advance one and withdraw the other.

Both methods gave the same result. It was possible to reduce, eliminate and even reverse the tendency of an animal to eat the food in a given right or left position. When the spatial advantage of both foods was the same, the conditions were most favorable for discovering the rat's food preferences. The animal could not, however, be forced into making a choice. Preference is assumed to depend upon organic factors rather than upon the environmental arrangement of test-foods.

Young's biographical note does not include a list of papers, but his Distinguished Scientific Contribution Award note does:

For his lifelong study of hedonic processes in behavior. Convinced of their significance for psychology, he endeavored to give objective reference and experimental validity to the concept. Although hedonic theorizing ran counter to the prevailing temper, he persisted in the belief that the control of behavior must be analyzed for affective value as well as intensity value. His research on preference showed the effect of experience in modifying acceptability; his work on need-free organisms clarified acceptance and appetitive behavior. Most recently, he has been examining composite stimuli and preference. Current renewed interest in hedonic theory rests in good measure on his experimental demonstrations and theoretical arguments."

(continued below)

20

The Decline Effect and the Scientific Method [link]

20

20

20

The Decline Effect and the Scientific Method [link]

20

20