= 27d567bc2d48d9f40e9ccc20ea370b15)
othercriteria comments on What are you working on? June 2012 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (75)
http://lesswrong.com/lw/9pl/automatic_programming_an_example/
Was this better?
I always want the shortest possible generating algorithm. Everything else, any "dimensionality" is just irrelevant.
Yes, I think that was better, because the ground truth is Kepler's third law and jimrandomh pointed out your method actually recaptures a (badly obfuscated and possibly overfit) variant of it.
"Dimensionality" is totally relevant in any approach to supervised learning. But it matters even without considering the bias/variance trade-off, etc.
Imagine that you have an high-dimensional predictor, of which one dimension completely determines the outcome and the rest are noise. Your shortest possible generating algorithm is going to have to pick out the relevant dimension. So as the dimensionality of the predictor increases, the algorithm length will necessarily increase, just for information-theoretic reasons.
How do you overfit Kepler's law?
edit: Retracted. I see now looking at the actual link the result wasn't just obfuscated but wrong, and so the manner in which it's wrong can overfit of course (and that matches the results).
To the extent that Kepler's laws are exact only for two-body systems of point masses (so I guess calling Kepler's third law the ground truth is a bit problematic) and to the extent that the data are imperfectly observed, there are residuals that over-eager models can try to match.
Edit: More generally, you don't overfit the underlying law, you overfit noisy data generated by the underlying law.
Kepler's law holds well. The influences of other planets are negligible for the precision we dealt with.
Dimensions irrelevant for the output, will fall out. Regardless if they are random or not. If they somehow (anyhow) contribute, their influence will remain in the evolved algorithm.
The simplest algorithm in the Kolmogorov's sense is the best you can hope for.