This principle is particularly important in statistical meta-analysis: because if you have a bunch of methodologically poor studies, each with small sample size, and then subject them to meta-analysis, what can happen is that the systematic biases in each study — if they mostly point in the same direction — can reach statistical significance when the studies are pooled.
Does anyone know how often this happens in statistical meta-analysis?
We can't know for certain. That's the idea of systematic biases. There no way to tell if all your trials are slanted in a specific fashion, if the biases also appears in your high quality studies.
On the other hand we have fields such as homeopathy or telephathy (Ganzfeld experiments) where there are meta-analysis that treat all studies mostly equally that find that homeopathy works and telepahty exist. On the other hand you have meta-analysis who try to filter out low quality studies who come to the conclusion that homeopathy doesn't work and telepathy doe...
Another month has passed and here is a new rationality quotes thread. The usual rules are:
And one new rule: