wedrifid comments on How Much Evidence Does It Take? - Less Wrong

34 Post author: Eliezer_Yudkowsky 24 September 2007 04:06AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (29)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: benelliott 15 February 2011 01:59:39PM *  9 points [-]

Lets do a check. Assume a worst case scenario where nobody publishes false results at all.

To get three p < 0.05 studies if the hypothesis is false requires on average 60 experiments. This is a lot but is within the realms of possibility if the issue is one which many people are interested in, so there is still grounds for scepticism of this result.

To get one p < 0.001 study if the hypothesis is false requires on average 1000 experiments. This is pretty implausible, so I would be much happier to treat this result as an indisputable fact, even in a field with many vested interests (assuming everything else about the experiment is sound).

Comment author: wedrifid 15 February 2011 02:06:18PM 1 point [-]

To get one p < 0.0001 study if the hypothesis is false requires on average 1000 experiments

One too many zeros in the p value there. The 1,000 figure matches p<0.001, which is also what Anders mentioned. (So your point is fine.)

Comment author: benelliott 15 February 2011 02:07:34PM 0 points [-]

Thanks