Houshalter comments on Friedman's "Prediction vs. Explanation" - Less Wrong

7 Post author: Eliezer_Yudkowsky 29 September 2008 06:15AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (79)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Houshalter 21 September 2013 10:42:59PM 2 points [-]

Imagine the ten experiments produced the following numbers as results: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

The first scientists hypothesis is this function: if n < 20 then n else 5, (where n is the number of the variable being tested in the experiment)

10 more experiments are done and of course it predicts the answers perfectly. Scientist two comes up with his hypothesis: n. That's it, just the value of n is the value that will be measured by the experiment.

Now, would you really trust the first hypothesis because it happened to have been made before the next ten experimental results were known?

In practice it's usually better to choose the hypothesis that has made more successful predictions in the past, because that is evidence that it isn't overfit. But complexity is also a more general way to keep overfitting in check. Useful when you can't just perform more experiments to test your hypothesises, or where it would be expensive to do so.