Cyan comments on Frequentist Statistics are Frequently Subjective - Less Wrong

59 Post author: Eliezer_Yudkowsky 04 December 2009 08:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (81)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 06 December 2009 01:45:27AM 0 points [-]

Releasing the data in dribs and drabs doesn't address this either.

It does force researchers into an ad hoc cross-validation scheme, doesn't it?

There's a difference between, on the one hand, having the data freely available and being intelligent enough to use cross-validation, and on the other, having someone paternalistically hold back the data from you.

If you start from the premise that researchers may fall into the overfitting trap, then you're already treating them adversarily. And if just one researcher overfitting a theory and so becoming irrefutable will screw everything up, then the paranoid approach to data release prevents that total cockup (at the cost of some interim inefficiencies, by hindering the responsible, good, researchers).

Comment author: Cyan 06 December 2009 01:53:01AM 0 points [-]

I'd rather wait for researchers to screw up and then hammer them.