jbay comments on Too good to be true - Less Wrong

24 Post author: PhilGoetz 11 July 2014 08:16PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (119)

You are viewing a single comment's thread. Show more comments above.

Comment author: Douglas_Knight 12 July 2014 09:21:48PM 4 points [-]

No, we are choosing the effect size before we do the study. We choose it so that if the true effect is zero, we will have a false positive exactly 5% of the time.

Comment author: jbay 17 July 2014 12:39:12PM *  2 points [-]

How does this work for a binary quantity?

If your experiment tells you that [x > 45] with 99% confidence, you may in certain cases be able to confidently transform that to [x > 60] with 95% confidence.

For example, if your experiment tells you that the mass of the Q particle is 1.5034(42) with 99% confidence, maybe you can say instead that it's 1.50344(2) with 95% confidence.

If your experiment happens to tell you that [particle Q exists] is true with 99% confidence, what kind of transformation can you apply to get 95% confidence instead? Discard some of your evidence? Add noise into your sensor readings?

Roll dice before reporting the answer?

Comment author: Douglas_Knight 17 July 2014 02:56:56PM 1 point [-]

We're not talking about a binary quantity.