Manfred comments on Too good to be true - Less Wrong

24 Post author: PhilGoetz 11 July 2014 08:16PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (119)

You are viewing a single comment's thread.

Comment author: Manfred 12 July 2014 12:50:51AM 4 points [-]

First thing, if you put something in your body, it has some effect, even if that effect is small. "No effect" results just rule out effects above different effect sizes (both positive and negative) with high probability, and there's no point talking about "a link" like it's some discrete thing (you sort of jump back and forth between getting this one right and wrong).

Second, different studies will rule out different effect sizes with 95% confidence - or to put it another way, at a given effect size, different studies will have different p-values, and so your probability exercise was pretty pointless because you didn't compare the studies' opinions about any particular effect size, just "whatever was 95%."

Third, I'd bet a nickel the effect sizes ruled out at 95% in all of these studies are well below the point where it would become concerning (like, say, the effect of the parents being a year older). That is, these studies all likely rule out a concerning effect size with probability much better than 95%.

Comment author: PhilGoetz 12 July 2014 01:38:10AM 3 points [-]

so your probability exercise was pretty pointless because you didn't compare the studies' opinions about any particular effect size

My probability exercise was not about effect size. It was about the probability of all studies agreeing by chance if there is in fact no link, and so the 95% confidence is what is relevant.

Third, I'd bet a nickel the effect sizes ruled out at 95% in all of these studies are well below the point where it would become concerning (like, say, the effect of the parents being a year older). That is, these studies all likely rule out a concerning effect size with probability much better than 95%.

Again, not relevant to the point I'm making here.

Comment author: Douglas_Knight 12 July 2014 08:03:02PM 4 points [-]

Not relevant to the main point that you're making, but relevant to your parenthetical:

(I was, as usual, irked that if a study failed to prove the existence of a link given various assumptions, it was usually cited as having shown that there was no link.)