Douglas_Knight comments on Frequentist Statistics are Frequently Subjective - Less Wrong

59 Post author: Eliezer_Yudkowsky 04 December 2009 08:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (81)

You are viewing a single comment's thread. Show more comments above.

Comment author: RichardKennaway 05 December 2009 10:49:25PM 6 points [-]

These seem relevant:

http://www.overcomingbias.com/2007/09/false-findings.html

"Why Most Published Research Findings Are False" I don't think he gives a figure of 2/3, but he does say: "It can be proven that most claimed research findings are false."

Comment author: Douglas_Knight 06 December 2009 04:00:28AM *  3 points [-]

"Why Most Published Research Findings Are False"

In fact, the origin of the statistic is an earlier paper by the same author, John PA Ioannidis, Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. Journalistic coverage of the later paper quotes a description of the earlier:

“Five out of six epidemiological studies have been contradicted in a very short period of time,” says Ioannidis, “while about one out of three randomized clinical trials were also refuted.”

Note that this is not 1/3 of all randomized studies, but just the highly cited ones, the ones that have any chance of an attempt at replication (and still 25% have no published attempt). It is not obvious to me whether these studies will be higher or lower quality than average. ETA: actually, there is a "control group" of articles published in the same prestigious journals, but with fewer citations. They are contradicted a little less often, p<.1, and not because people aren't bothering to replicate them.

Comment author: ChristianKl 06 December 2009 11:20:06PM 3 points [-]

But there still quite a difference between claiming that 1/3 or 2/3 of randomized studies are wrong.

Comment author: Douglas_Knight 07 December 2009 12:44:27AM 1 point [-]

But there still quite a difference between claiming that 1/3 or 2/3 of randomized studies are wrong.

Yes, the 2/3 is probably an error on Eliezer's part. He doesn't say randomized, so he could be misremembering the 5/6, but the 5/6 is probably not the figure to quote and Ioannidis is a little misleading in putting it first. Only 6 of the 45 highly-cited studies were retrospective and I think that is representative of medical research in the prestigious journals. (the discussion of the control group should address this, but I don't see it)