Cyan comments on Frequentist Statistics are Frequently Subjective - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (81)
It does force researchers into an ad hoc cross-validation scheme, doesn't it?
If you start from the premise that researchers may fall into the overfitting trap, then you're already treating them adversarily. And if just one researcher overfitting a theory and so becoming irrefutable will screw everything up, then the paranoid approach to data release prevents that total cockup (at the cost of some interim inefficiencies, by hindering the responsible, good, researchers).
I'd rather wait for researchers to screw up and then hammer them.