nshepperd comments on Too good to be true - Less Wrong

24 Post author: PhilGoetz 11 July 2014 08:16PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (119)

You are viewing a single comment's thread. Show more comments above.

Comment author: kilobug 12 July 2014 02:17:33PM 5 points [-]

I don't think the "95% confidence" works that way. It's a lower bound, you never try to publish anything with a lower than 95% confidence (and if you do, your publication is likely to be rejected), but you don't always need to have exactly 95% (2 sigma).

Hell, I play enough RPGs to know that rolling 1 or 20 in a d20 is frequent enough ;) 95% is quite low confidence, it's really a minimum at which you can start working, but not something optimal.

I'm not sure exactly in medicine, but in physics it's frequent to have studies at 3 sigma (99.7%) or higher. The detection of the Higgs boson by the LHC for example was done within 5 sigma (one chance in a million of being wrong).

Especially in a field with high risk of data being abused by ill-intentioned people such as "vaccine and autism" link, it would really surprise me that everyone just kept happily the 95% confidence, and didn't aim for much higher confidence.

Comment author: nshepperd 12 July 2014 02:26:47PM 12 points [-]

5 sigma (one chance in a million of being wrong)

Careful! That's a one chance in a million of a fluke occuring (given the null hypothesis). Probability of being wrong is P(~H1 | 5 sigma) rather than P(5 sigma | H0), and on the whole unmeasurable. :)