Desrtopa comments on How Much Evidence Does It Take? - Less Wrong

34 Post author: Eliezer_Yudkowsky 24 September 2007 04:06AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (29)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: Desrtopa 01 April 2012 03:48:03PM *  2 points [-]

This is assuming proper methodology and statistics so that the p-value actually matches the chance of the result arising by chance. In practice, since even your best judgment of the methodology is not going to account for certainty in the soundness of the experiment, I would say that a p-value of 0.001 constitutes considerably less than 10 bits of evidence, because the odds that something was wrong with the experiment are better than the odds that the results were coincidental. Multiple experiments with lower cumulative p-value can still be stronger evidence if they all make adjustments to account for possible sources of error.