Bioinformatics is a neat example of how this bias can arise. Forgive me: I'm going to go into excessively nerdy detail about a specific example, because bioinformatics is cool.

Suppose that a biologist has amino acid sequences for 100 species' versions of the same protein. The species are not closely related, and the protein sequences for each species have a lot of variation between them. The biologist wants to find parts of the protein sequence that have remained similar for a long evolutionary time. The usual way to do this is to try to line up matching parts of the sequences, inserting gaps and accepting mismatches where necessary.

Aligning multiple sequences is a very hard problem, computationally, so we have to use approximate methods. The most common way is to break it down to a problem we can solve much more easily: aligning two sequences, then computing an average sequence for that clump which can be used to add another sequence to it. And another, and another. These algorithms compare all pairs of sequences to measure how similar they are, and then starts clumping together similar sequences in a tree that looks a lot like a diagram of evolutionary ancestry. At the end, your sequences should be aligned acceptably. You hope.

Of course, this assumes that some of the sequences are more closely related than others, and that you can form a nice tree shape. And it's very approximate, and there's lots of opportunity for error to creep in. So for some data, this works great, and for some data, it gives nonsense. Another method looks for small, very common subsequences, and iteratively refines the alignment based on these. Again, this works great for some data, and not so well for others. And then of course there are dozens of other methods, based on things like genetic algorithms, or simulated annealing, or hidden Markov models, and all of these have times when they work well and times when they don't.

So what does a biologist do? Try several methods, of course! The Right Way to do this is to run the same input through a bunch of algorithms, check them for agreement, see if several algorithms are giving the same results, and then apply any biological knowledge you may have to help decide what's working. The way that I'm sure some people end up using is that they reach for whatever multiple sequence alignment program they like most, run it, and trust its output if it's not too surprising. If it surprises them, they might try another algorithm. If that produces something more like what they were expecting to see, they may then stop, because they're busy, damn it, and they have work to do. Most biologists really don't want to be computer scientists, and it's tempting to treat the computer as a magic answer machine. (The obvious solution here is to rent some computing cloud time and run a bunch of different algorithms on your data every time you do multiple sequence alignment, and have your software automatically compare the results. More dakka helps.)

I don't know how common this sort of error is in practice, but there's certainly the potential for it.

This happens in other ways also. Mathematicians are much more likely to look suspiciously at a proof of a surprising result than at a proof of a result that seems very reasonable.

Most of the NLP (natural language processing, not the other NLP) research I do is loosely validated by a shared task and competitive performance on held-out test data. There is not much chance that a bug leads to higher task accuracy (usually agreement with human judgments). But it's true that if you had a great idea which could lead to a huge improvement, but only shows a small one because your implementation has bugs, then you may assume that the idea was not really that effective, and not hunt for the bugs. Whereas if the performance is actually worse than before this change, and you know the change to be a good idea, you will hunt very hard for the bugs.

I suppose wherever progress is most incremental (everyone is using the same basic systems with the novel part being some small addition), then there's also a real risk of bad or buggy small changes being published because they happen by pure chance to give a slightly higher accuracy on the standard test data. But such random-noise additions will in fact do worse on any different test set, and there is some chance of detecting them using significance tests.

I guess it is a problem for the field that new (different domain or merely freshly generated) test sets aren't automatically used to score all published past systems to detect that effect. This should theoretically be possible by requiring open-source implementations of published results.

I completely agree that debugging processes and confirmation biases interact (and I've seen the effect enough times in practice to understand that my first three hypothesis iterations about nearly any aspect of a giant data set will probably be wrong enough to detect if I try), but for the sake of debiasing our thinking about debiasing techniques, I wonder about opinions about things we should expect to help rather than hurt?

For example, I find sometimes that I create pipelines of data where there will be a few gigs stored at each stage with less than a hundred lines of code between each stage. If I test the data at each stage in rather simple ways (exploring outliers, checking for divide by zero, etc) that I gain additional "indirect code verification" by noting that despite the vast space of possible output data with 2^10^10 degrees of freedom, my scripts are producing nothing that violates a set of filters that would certainly be triggered by noise and "probably" (craft and domain knowledge and art certainly goes into this estimate) be triggered by bugs.

A summary of these experiences would be something like "the larger the data set the less likely that bugs go undetected". And programmable computers do seem to enable giant data sets... so...

I'm not sure I have enough information to say whether the expected rate of bug-based confirmations should be 20% or 90% when certain kinds of software or coding techniques are involved, but it seems like there can be other "big picture factors" that would reduce biases.

Do you agree that larger data sets create more room to indirectly verify code? Can you think of other factors that would also push scientific programming in a positive direction?

This is a good point - and I would add that most of the errors that have this effect are not coding errors, but conceptual errors. A good practice, before performing an experiment, is to ask yourself, "What would I think if this just didn't work?" And try, hard, to come up with a justification for it not working. Sometimes I will realize there's an additional factor that may cause unexpected results.

Most of the time, though, I can't think how the experiment could fail until it actually fails. And then it seems obvious in retrospect why it failed.

neq1, you have somehow managed to provoke the most informative set of comments I've seen on LW in awhile. Everyone's giving interesting details about their own research. Awesome! Upvoted just for that. (Though the post is fine, too.) And cheers to the commenters too - this is the way life should be like.

Tangentially related: Phil Goetz's Computer bugs and evolution (with further details in the comments).

Also, your link to Wikipedia's page on confirmation bias has a typo.

Accidentally promoted this whilst trying to add an "end summary" bar. Demoted thereafter. No offense meant, sorry for the inconvenience.

Write several pieces of analysis code, ideally in different languages, and check that the results are the same? Even better, have someone else replicate your analysis code. That way you have a somewhat independent source of confirmation.

Also, use practices like tons of unit testing which minimize the chance for bugs in your code. All this must be done before you see the results, of course.

Is this confirmation bias really that bad in practice? Scientists get credit for upsetting previous consensus. So this may lead potentially disruptive research to happen slightly less often. But it remains the case that an attempted change to the consensus - a "surprising" result will still get changed eventually, by someone who doesn't question the surprising result, or questions it but thoroughly reviews their code and stands by it. So evidence for change will come slightly less often than it could, but changes will still be correct. Doesn't seem like a big deal.

Science got the charge on an electron right, even after Milliken's mistake.

Different fields have different standards. Some are more exacting than others and require full release of source code, have standard hidden-data competitions, have a culture of reviewing the software and attempting to reproduce &c (not all of the above is applicable to all fields). Others, not so much: you publish by giving a high level description of something you coded and people believe that you did it correctly and didn't spend hours looking for the parameters that gave you the prettiest graph. Debugging by "hacking 'til the graph is publishable" is, unfortunately, too common in some of those fields.

Many scientists are completely unaware of anything other than their field and will claim that "this is the it's done" whilst they only mean "this is the way that people in my narrow sub-field do it if they want to get published".