Who has not experienced the chilling memory of the better things? How it creeps over the spirit of one's current dreams! Like the specter at the banquet it stands, its substanceless eyes viewing with a sad philosophy the make-shift feast.

-Theodore Dreiser, The Titan

Wow, I just read Robin's writeup on this and it caused me to significantly lower the amount of credence I place on his other positions (but very slightly lower my opinion of supplements). It just struck me as overwhelmingly sloppy and rhetorical. Particularly his justification attempt in response to this thread. (But I suppose Robin's responses to criticism have never impressed me anyway.)

It would be nice if the top scoring all-time posts really reflected their impact. Right now there is some bias towards newer posts. Plus, Eliezer's sequences appeared at OB first, which greatly reduced LW upvotes.

Possible solution: every time a post is linked to from a new post, it gets an automatic upvote (perhaps we don't count it if linked to by same author). I don't know if it's technically feasible

I suppose what I was getting at was asking whether it is something that you have enough interest in or ideas about that you would like to collaborate on.

Rather tangentially...
Here is a linguistic danger of your article. It's not much of a danger if one reads the text in order, but if one starts with headlines, "How common are coding errors in research?" could be confusing because many fields of research (probably most researchers) use "coding" to mean "labeling." eg, when a human watches a tape of a monkey and codes it as "looked left" or "looked right." And they keep track of rates of "coding errors."
So I'd suggest changing that headline to "programming."

I know of no such study, and have failed to find one in a quick literature search.

I occasionally run behavioral psych studies, and this seems like a good candidate. How would you feel about me adapting this into a study?

In the first example, you couldn't play unless you had at least 100M dollars of assets. Why would someone with that much money risk 100M to win a measly 100K, when the expected payoff is so bad?

I would not be surprised if at least 20% of published studies include results that were affected by at least one coding error.

My intuition is that this underestimates the occurrence, depending on the field. Let us define:

• CE = study has been affected by at least one coding error
• SP = study relies on a significant (>500 LOC) amount of custom programming

Then I'd assign over 80% to P(CE|SP).

My mom is a semi-retired neuroscientist, she's been telling me recently how appalled she's been with how many researchers around her are abusing standard stats packages in egregious ways. The trouble is that scientists have access to powerful software packages for data analysis but they often lack understanding of the concepts deployed in the packages, and consequently make absurd mistakes.

"Shooting yourself in the foot" is the occupational disease of programmers, and this applies even to non-career programmers, people who program as a secondary requirement of their job and may not even have any awareness that what they're doing is programming.

Feynman once talked about this specific issue during a larger speech:

We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off, because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of the electron, after Millikan. If you plot them as a function of time, you find that one is a little bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn't they discover that the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease.