LESSWRONG
LW

32

[ Question ]

Hard to find factors messing up experiments: Examples?

15th Nov 2019

2 min read

32

In Richard Feynman's talk on cargo cults he mentions a story about experiments with rat mazes (skip if you recognize it)

All experiments in psychology are not of this type, however. For example, there have been many experiments running rats through all kinds of mazes, and so on—with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.

The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and, still the rats could tell.

He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell.

I also recall some experiment where it was eventually discovered that the control of some microbiology experiment was screwed because the paper towels used in some process were made with wood that was treated with a particular chemical before it was turned into the paper towel (was this linked by gwern at some point?).

I'm interested in collecting any and all examples you have of these events; through painstaking effort someone realized that XYZ was interfering with an intended experiment (extra points if XYZ was completely outside the experimenters initial frame of "factors that might be relevant to this experiment") . Links to source if possible.

Practice & Philosophy of ScienceWorld Modeling

32

Hard to find factors messing up experiments: Examples?

8johnswentworth

26noggin-scratcher

New Answer

New Comment

2 Answers sorted by
top scoring

Nov 16, 2019

80

A few stories from my undergrad (some firsthand, some secondhand):

One of the standard experiments in the undergrad intro physics lab involved a pendulum. One professor would loosen the fasteners on the top before the lab began. When students got confusing results, he would ask them what could be the cause, and they would stand there scratching their heads as the bar to which the pendulum was attached went clanging back and forth.
During one professor's time at grad school, a voltage probe was showing bizarre oscillations of tens of volts. After some investigation, it turned out that the "ground" ran down through a pipe into the local groundwater, and a long wire elsewhere in the system ran over the ground for a ways, and the whole thing formed a giant antenna. They were picking up radio waves.
No undergrad lab involving an oscilloscope would be complete without someone noticing mysterious oscillations in their circuit. Usually these turn out to be oscillations at 60 Hz - anywhere urban or indoors is flooded with 60 Hz waves from the power lines & outlets.
While calibrating a pH sensor, I noticed that the supposedly-deionized water was off from where it should be. Our lab manager thought it was the local air quality (we were in LA). Apparently there's a fair bit of variance in the pH of DI water exposed to air depending on where you are.

Also, as a source of more of these stories, you might check out the classic experiments measuring various physical constants - gravitational constant, electron mass & charge, etc. Usually they involve a whole series of tricks to control for various error sources.

Nov 16, 2019

60

A friend told me this story many years ago: He was working on repairing some electronic staff and one block had a green light turned on when the block was turned off by a switch, but not disconnected from the power line. However, there are no short circuits in it. After long investigation, mostly for curiosity, he found that some piece of alloy covered another piece and together they created a capacitor, which was able to let in AC part of incoming signal and power the light in the gadget.

11 comments, sorted by

Click to highlight new comments since: Today at 10:46 PM

[-]noggin-scratcher5y260

The "Phantom of Heilbronn" was an unknown female serial killer, whose DNA was found at numerous crime scenes across Austria, France and Germany from 1993 to 2009. It was eventually realised that the DNA belonged to a woman working in a factory, making the cotton swabs that were used to collect evidence.

Thanks! This is an okay article that adds more exposition, and it looks like this incident spawned a new protocol for avoiding swab contamination.

Ha,I think I remember seeing a CSI episode based on this story but never realised it was based on a true story. At the time I thought the plot was a tad implausible but that shows what I know!

[-]ErickBall5y160

Not an experiment but similar situation: I talked with a scientist at NIST who told me that their nuclear reactor was shut down once for maintenance, and while the core was out they had the cleaning staff come in and scrub the aluminum panels that lined the walls, since they hadn't been cleaned in years. When the maintenance was done, the reactor wouldn't start back up. Eventually they figured out that the cleaning spray used on the walls contained borax. Boron is a potent neutron absorber and the trace amounts left behind were keeping the reactor from going critical. They had to toss out all the panels and replace them.

I am curious about the last bit in the post: " thought to be a completely irrelevant factor, was wreaking havoc on an experiment."

When saying "thought to be" are you suggesting that perhaps we should not view some of these factors as irrelevant and messing up results or should I see the emphasis on "the painstaking effort" applied to identify some irrelevant factor that messed things up?

More on the painstaking effort. I was gesturing at the fact that Young probably didn't think the floors had anything to do with running rat mazes at the beginning, and later it turned out to be important. Through painstaking effort he realized this was what was messing stuff up.

While perhaps not directly pertinent to the goals of collecting these data points, I wonder if it's accurate to say the rat example is really a case of "completely irrelevant" factors. Seems to me the lesson from the rat experiment is rats prefer navigating via sounds over images. Taking away their ability to hear where they are is not training them to rely on vision but forcing them to do so.

It's hard to say because no one has tracked down if the rat story happened; although we did find some instances which looked real of very similar stories.

"completely irrelevant"

Seemingly irrelevant

Maybe, and this is getting to be a divergence off on a tangent. I wonder, for the rat experiment, if the sounds were actually rather relevant and not something that should have been just excluded.

I think I see what you mean. Discovering rats can navigate mazes via sounds is a new interesting thing. You could go on to study more about how that works and how it interacts with other things rats do. Though if you just wanted to look at a some aspect of how rats learn, you'd have to account for the fact that in your experiment their hearing could circumvent some aspect of your design.