gwern comments on Frequentist Statistics are Frequently Subjective - Less Wrong

59 Post author: Eliezer_Yudkowsky 04 December 2009 08:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (81)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 05 December 2009 06:51:21PM 8 points [-]
Comment author: Eliezer_Yudkowsky 05 December 2009 08:21:49PM 2 points [-]

I also remember seeing this on OB but was having trouble searching for some reason.

Comment author: RichardKennaway 05 December 2009 10:49:25PM 6 points [-]

These seem relevant:

http://www.overcomingbias.com/2007/09/false-findings.html

"Why Most Published Research Findings Are False" I don't think he gives a figure of 2/3, but he does say: "It can be proven that most claimed research findings are false."

Comment author: Douglas_Knight 06 December 2009 04:00:28AM *  3 points [-]

"Why Most Published Research Findings Are False"

In fact, the origin of the statistic is an earlier paper by the same author, John PA Ioannidis, Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. Journalistic coverage of the later paper quotes a description of the earlier:

“Five out of six epidemiological studies have been contradicted in a very short period of time,” says Ioannidis, “while about one out of three randomized clinical trials were also refuted.”

Note that this is not 1/3 of all randomized studies, but just the highly cited ones, the ones that have any chance of an attempt at replication (and still 25% have no published attempt). It is not obvious to me whether these studies will be higher or lower quality than average. ETA: actually, there is a "control group" of articles published in the same prestigious journals, but with fewer citations. They are contradicted a little less often, p<.1, and not because people aren't bothering to replicate them.

Comment author: ChristianKl 06 December 2009 11:20:06PM 3 points [-]

But there still quite a difference between claiming that 1/3 or 2/3 of randomized studies are wrong.

Comment author: Douglas_Knight 07 December 2009 12:44:27AM 1 point [-]

But there still quite a difference between claiming that 1/3 or 2/3 of randomized studies are wrong.

Yes, the 2/3 is probably an error on Eliezer's part. He doesn't say randomized, so he could be misremembering the 5/6, but the 5/6 is probably not the figure to quote and Ioannidis is a little misleading in putting it first. Only 6 of the 45 highly-cited studies were retrospective and I think that is representative of medical research in the prestigious journals. (the discussion of the control group should address this, but I don't see it)

Comment author: Eliezer_Yudkowsky 05 December 2009 11:31:10PM 1 point [-]

Thanks. Since I can't find the original reference (if, indeed, it ever existed outside my imagination) these will do equally well to illustrate the point.

Comment author: ChristianKl 06 December 2009 11:20:14PM 7 points [-]

It's ironic that you use a number for which you don't know your source in an essay to recommend that public always release all their data.

Living by one's own standards is always hard when you are a skeptic ;)