To the extend that's true careful replication of studies to identify factors is important if you don't want to practice what Feymann described as Cargo Cult science. If you follow Feymann argument physicists also would get a bunch of bad results if they would work with the scientific standards used in psychology.

Feymann on rat psychology:

All experiments in psychology are not of this type, however. For example, there have been many experiments running rats through all kinds of mazes, and so on--with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.

The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell.

He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell.

Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers that clues that the rat is really using-- not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running.

I looked up the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running the rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about the rats.

Nutrition is really a different case than a lot of psychology. There are question in psychology such as whether doing certain things to a child in it's childhood effect whether that child is a healthy adult. Those questions are hard to investigate scientifically because of time lag. The same isn't true for many psychology experiments.

The number of confounding factors isn't that important if it's possible to do controlled experiments that control for them. Nutrition science has the problem that you usually can't do good controlled experiments or those are very expensive.

I'm Tom, 23 year old uni drop out (existential apathy is a killer), majored in Bioscience for what its worth. Saw the name of this site while browsing tvtropes and was instantly intrigued, as "less wrong" has always been something of a mantra for me. I lurked for a while and sampled the sequences and was pleased to note that many of the points raised were ideas that had already occurred to me.

Its good to find a community devoted to reason and that seems to actually think where most people are content not to. I'm looking forward suckling off the collective wisdom of this community, and hopefully make a valuable contribution or two of my own.

If you want to know whether food is good or bad you have to look at mortality which means you might have to wait a decade.

A lot of psychology experiments claim effects over much shorter timeframes.