DanielLC comments on Too good to be true - Less Wrong

24 Post author: PhilGoetz 11 July 2014 08:16PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (119)
Tags: publication literature 95%

Comments (119)

You are viewing a single comment's thread. Show more comments above.

Comment author: mwengler 12 July 2014 04:40:06PM *  12 points [-]

Presumably, with 95% confidence.

Presumably? I checked the definition of presumably:

used to convey that what is asserted is very likely though not known for certain.

So you take this uncertain confidence level of 95% and find:

What are the odds, supposing there is no link between X and Y, of conducting 60 studies of the matter, and of all 60 concluding, with 95% confidence, that there is no link between X and Y?

Answer: .95 ^ 60 = .046. (Use the first term of the binomial distribution.)

OK so you presumed 95% confidence level and showed that that confidence level is inconsistent with unanimity across 60 studies.

Assuming the studies are good, what confidence level would be consistent with unanimity?

Answer: .99^60 = 54%

So from this result we conclude either 1) there is a a problem with at least some of the studies or 2) there is a problem with the presumption of 95% confidence level, but a 99% confidence level would work fine.

For this post to have positive value, the case for picking only conclusion 1 above, and not considering conclusion 2, needs to be made. If the 95% confidence level is in fact EXPLICIT in these studies, then that needs to be verified, and the waffle-word "presumably" needs to be removed.

Comment author: DanielLC 20 July 2014 05:58:39AM 2 points [-]

That part was just him noticing his confusion. The only way to figure out what the real confidence levels were would be to try and find the studies, which is exactly what he did.

Comment author: private_messaging 20 July 2014 09:47:58AM *  4 points [-]

I read his post twice and I still don't see him having figured out the real confidence levels or claiming to have.

edit: besides, Phil's own claims don't even meet the 95% confidence, and god only knows out of how big of a pool he fished this bias example from, and how many instances of 'a few studies find a link but most don't' he ignored until he came up with this.

Powered by Reddit Powered by Reddit