RichardKennaway comments on Open thread, July 23-29, 2013 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (197)
Here's a recent article by Andrew Gelman on how easy it is to find and publish completely meaningless "significant" results.
The key concept here is "researcher degrees of freedom": all of the things the researcher could have tested, but didn't, or did, but didn't bother to mention as there was no p<0.05 result.
The money quote (albeit buried at the very end of the article):
This is strong stuff too:
On his blog he calls this "the scientific mass production of spurious statistical significance".
If you assume regression to the mean will occur in replication, you should only pay attention to fairly large effect sizes. This has a way of washing out a lot of the noise in areas where previously it seemed hard to draw a conclusion.