= 27d567bc2d48d9f40e9ccc20ea370b15)
gwern comments on Open Thread, November 8 - 14, 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (141)
This seems like a big deal:
http://www.pnas.org/content/early/2013/10/28/1313476110.full.pdf
Basically, dude illustrates equivalence between p-values and Bayes factors and concludes that 17-25% of studies with a p-value acceptance threshold of 0.05 will be wrong. This implies that the lack of reproducibility in science isn't necessarily due to egregious misconduct, etc., but rather insufficiently strict statistical standards.
So is this new/interesting, or do I just naively think so because it's not my field?
Not a big deal. The estimate you're impressed by can be done from power and prior odds like in Ioannides's famous paper and are similar to Leek's estimates from p-value distributions, and the recommendations baffle me - increase alpha?! P-value hacking is part of how we got here in the first place!
Is there a lower hanging fruit you have in mind?
I don't know any easy solutions to the low replication rate of many areas right now. It seems to be fundamentally a systematic problem of incentives. Even the easiest and most basic remedies like clinical trial registries are not being enforced, so it's hopeless to expect reforms like making all studies well-powered. I do think that increasing alpha is unlikely to fix the problems and is likely to backfire by making things worse and rewarding cheaters & punishing honest researchers: the smaller the p-value required, the more you reward people who can run hundreds of analyses to get a p-value under the threshold and the more you punish honest researchers who did one analysis and stuck with it.