= 27d567bc2d48d9f40e9ccc20ea370b15)
gwern comments on Too good to be true - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (119)
Presumably? I checked the definition of presumably:
So you take this uncertain confidence level of 95% and find:
OK so you presumed 95% confidence level and showed that that confidence level is inconsistent with unanimity across 60 studies.
Assuming the studies are good, what confidence level would be consistent with unanimity?
Answer: .99^60 = 54%
So from this result we conclude either 1) there is a a problem with at least some of the studies or 2) there is a problem with the presumption of 95% confidence level, but a 99% confidence level would work fine.
For this post to have positive value, the case for picking only conclusion 1 above, and not considering conclusion 2, needs to be made. If the 95% confidence level is in fact EXPLICIT in these studies, then that needs to be verified, and the waffle-word "presumably" needs to be removed.
Is there any reason at all to think that these medical studies didn't use 95%? The universal confidence level, used pretty much everywhere in medicine and psychology except in rare subfields like genomics, so universal that authors of papers typically don't even bother to specify or justify the confidence level?
I'd love to see a paper get published that justified the confidence level with "because if I wanted to do rigorous science I would have studied physics" or "because we only have enough jelly beans to run 30 studies, will only be given more jelly beans if we get a positive result and so need to be sure".
There's all sorts of things one has to control for, e.g. parent's age, that may inflate the error bars (if the error in imperfectly controlling for a co-founder is accounted for), putting zero within the error bars. Without looking at all the studies one can't really tell.
Some studies ought to also have a chance of making a superfluous finding that 'vaccines prevent autism', but apparently that was not observed either.
What does that have to do with whether the researchers followed the nigh-universal practice of setting alpha to 0.05?
Example: I am measuring radioactivity with a Geiger counter. I have statistical error (with the 95% confidence interval), but I also have systematic error (e.g. the Geiger counter's sensitivity is 'guaranteed' to be within 5% of a specified value). If I am reporting an unusual finding, I'd want the result not to be explainable by the sum of statistical error and the bound on the systematic error. Bottom line is, generally there's no guarantee that "95% confidence" findings will go the other way 5% of the time. It is perfectly OK to do something that may inadvertently boost the confidence.
Suppose there were 60 studies that showed no correlation between autism and vaccines at a 99% confidence level. THen it would not be particularly surprising that there were indeed 60 studies with that result.
Would you expect the authors to point out that their result was actually 99% confident even though the usual standard, which they were not explicitly claiming anyway, was 95%?
retracted