PhilGoetz comments on Error detection bias in research - Less Wrong

54 Post author: neq1 22 September 2010 03:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (36)

You are viewing a single comment's thread.

Comment author: PhilGoetz 24 September 2010 04:03:24PM 2 points [-]

This is a good point - and I would add that most of the errors that have this effect are not coding errors, but conceptual errors. A good practice, before performing an experiment, is to ask yourself, "What would I think if this just didn't work?" And try, hard, to come up with a justification for it not working. Sometimes I will realize there's an additional factor that may cause unexpected results.

Most of the time, though, I can't think how the experiment could fail until it actually fails. And then it seems obvious in retrospect why it failed.