Desrtopa comments on Rationality Quotes April 2014 - LessWrong

8 Post author: elharo 07 April 2014 05:25PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (656)

You are viewing a single comment's thread. Show more comments above.

Comment author: raisin 01 April 2014 04:09:29PM 7 points [-]

This principle is particularly important in statistical meta-analysis: because if you have a bunch of methodologically poor studies, each with small sample size, and then subject them to meta-analysis, what can happen is that the systematic biases in each study — if they mostly point in the same direction — can reach statistical significance when the studies are pooled.

Does anyone know how often this happens in statistical meta-analysis?

Comment author: Desrtopa 01 April 2014 04:21:11PM 6 points [-]

As a percentage? No. But qualitatively speaking, "often."

The most recent book I read discusses this particularly with respect to medicine, where the problem is especially pronounced because a majority of studies are conducted or funded by an industry with a financial stake in the results, with considerable leeway to influence them even without committing formal violations of procedure. But even in fields where this is not the case, issues like non-publication of data (a large proportion of all studies conducted are not published, and those which are not published are much more likely to contain negative results) will tend to make the available literature statistically unrepresentative.