Hello, Less Wrong!
This seems like a community with a relatively high density of people who have worked in labs, so I'm posting here.
I recently finished the first draft of something I'm calling "The Hapless Undergraduate's Guide to Research" (HUGR). (Yes, "HUGS" would be a good acronym, but "science" isn't specific enough.) Not sure if it will ever be released, or what the final format will be, but I'll need more things to put in it whatever happens.
Basically, this is meant to be an ever-growing collection of mistakes that new researchers (grad or undergrad) have made while working in labs. Hundreds of thousands of students around the English-speaking world do lab work, and based on my own experiences in a neuroscience lab, it seems like things can easily go wrong, especially when rookie researchers are involved. There's nothing wrong with making mistakes, but it would be nice to have a source of information around that people (especially students) might read, and which might help them watch out for some of the problems with the biggest pain-to-ease-of-avoidance ratios.
Since my experience is specifically in neuroscience, and even more specifically in "phone screening and research and data entry", I'd like to draw from a broad collection of perspectives. And, come to think of it, there's no reason to limit this to research assistants--all scientists, from CS to anthropology, are welcome!
So--what are some science mistakes you have made? What should you have done to prevent them, in terms of "simple habits/heuristics other people can apply"? Feel free to mention mistakes from other people that you've seen, as long as you're not naming names in a damaging way. Thanks for any help you can provide!
And here are a couple of examples of mistakes I've gathered so far:
--Research done with elderly subjects. On a snowy day, the sidewalk froze, so subjects couldn't be screened for a day, because no one thought to salt the sidewalks in advance. Lots of scheduling chaos.
--Data entry being done for papers with certain characteristics. Research assistants and principal investigator were not on the same page regarding which data was worth collecting. Each paper had to be read 7 or 8 times by the time all was said and done, and constructing the database took six extra weeks.
--A research assistant clamped a special glass tube too tight, broke it, and found that replacements would take weeks to come in... well, there may not be much of a lesson in that, but maybe knowing equipment is hard to replace cold subconsciously induce more caring.
Not taking account of multiplicity.
Ideally you should plan exactly how you're going to analyse the data before the experiment but in reality students muddle through a bit.
analyzing data in multiple ways is a big no-no if you're just hunting for that elusive 0.05 P value to get it published.
It's stupid and causes statisticians to tear their hair out(both the arbitrary requirement a lot of journals set and the bad stats by researcher) but it's the reality in a lot of research.
Doing that can be compensated for as long as you keep track of what you tried and make that data available.
It's even worse because often people, including experienced professors, delude themselves with bad stats and waste time and money chasing statistical phantoms because they went significance mining.