hyporational comments on Open Thread, November 8 - 14, 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (141)
Not a big deal. The estimate you're impressed by can be done from power and prior odds like in Ioannides's famous paper and are similar to Leek's estimates from p-value distributions, and the recommendations baffle me - increase alpha?! P-value hacking is part of how we got here in the first place!
Is there a lower hanging fruit you have in mind?
I don't know any easy solutions to the low replication rate of many areas right now. It seems to be fundamentally a systematic problem of incentives. Even the easiest and most basic remedies like clinical trial registries are not being enforced, so it's hopeless to expect reforms like making all studies well-powered. I do think that increasing alpha is unlikely to fix the problems and is likely to backfire by making things worse and rewarding cheaters & punishing honest researchers: the smaller the p-value required, the more you reward people who can run hundreds of analyses to get a p-value under the threshold and the more you punish honest researchers who did one analysis and stuck with it.