If the ellipse is very narrow, things are indeed well-modeled by a linear relationship, and the biggest Y coordinate for a point is likely to also have close to biggest X coordinate.

If the ellipse is not narrow, that could be for two reasons. Either the underlying truth is indeed linear, but your data is very noisy. Or the underlying truth is not linear, and you should not use a linear model. (Or both, naturally).

If the underlying truth is linear, but your data is very noisy, then what happens to the X coordinate of points with given Y values is mostly determined by the noise.

If the underlying truth is not linear, why should we expect sensible answers from a linear model?

I don't follow you. Overfitting happens when your model has too many parameters, relative to the amount of data you have. It is true that linear models may have few parameters compared to some non-linear models (for example linear regression models vs regression models with extra interaction parameters). But surely, we can have sparsely parameterized non-linear models as well.

All I am saying is that if things are surprising it is either due to "noise" (variance) or "getting the truth wrong" (bias). Or both.

I agree that "models we can quickly and easily use while under publish-or-perish pressure" is an important class of models in practice :). Moreover, linear models are often in this class, while a ton of very interesting non-linear models in stats are not, and thus are rarely used. It is a pity.

Comment author:henry4k2PH4
04 August 2014 12:40:26AM
2 points
[-]

A technical difficulty with saying that overfitting happens when there are "too many parameters" is that the parameters may do arbitrarily complicated things. For example they may encode C functions, in which case a model with a single (infinite-precision) real parameter can fit anything very well! Functions that are linear in their parameters and inputs do not suffer from this problem; the number of parameters summarizes their overfitting capacity well. The same is not true of some nonlinear functions.

To avoid confusion it may be helpful to define overfitting more precisely. The gist of any reasonable definition of overfitting is: If I randomly perturb the desired outputs of my function, how well can I find new parameters to fit the new outputs? I can't do a good job of giving more detail than that in a short comment, but if you feel confused about overfitting, here's a good (and famous) article about frequentist learning theory by Vladimir Vapnik that may be useful:

Comment author:IlyaShpitser
04 August 2014 04:48:38PM
*
2 points
[-]

This is about "reasonable encoding" not "linearity," though. That is, linear functions of parameters encode reasonably, but not all reasonable encodings are linear. We can define a parameter to be precisely one bit of information, and then ask for the minimum of bits needed.

I don't understand why people are so hung up on linearity.

Comment author:[deleted]
01 August 2014 11:39:14AM
2 points
[-]

I don't follow you. Overfitting happens when your model has too many parameters, relative to the amount of data you have. It is true that linear models may have few parameters compared to some non-linear models (for example linear regression models vs regression models with extra interaction parameters). But surely, we can have sparsely parameterized non-linear models as well.

Sure, technically if Alice fits a small noisy data set as y(x) = a*x+b and Bob fits it as y(x) = c*Ai(d*x) (where Ai is the Airy function) they've used the same number of parameters, but that won't stop me from rolling my eyes at the latter unless he has a good first-principle reason to privilege the hypothesis.

The problem is more practical than theoretical (don't have the links to hand. but you can find some in my silos of expertise post). Statisticians do not adjust properly for extra degrees of freedom, so among some category of published models, the linear ones will be best. Also, it seems that linear models are very good for modelling human expertise - we might think we're complex, but we behave pretty linearly.

I still don't understand your original "because." I am talking about modeling the truth, not modeling what humans do. If the truth is not linear and humans use a linear modeling algorithm, well then they aren't a very good role model are they?

Because human flaws creep in in the process of modelling as well. Taking non linear relationships into account (unless there is a causal reason to do so) is asking for statistical trouble unless you very carefully account for how many models you have tested and tried (which almost nobody does).

Comment author:Lumifer
30 July 2014 02:41:34PM
4 points
[-]

Taking non linear relationships into account (unless there is a causal reason to do so) is asking for statistical trouble unless you very carefully account for how many models you have tested and tried (which almost nobody does).

First, the structure of your model should be driven by the structure you're observing in your data. If you are observing nonlinearities, you'd better model nonlinearities.

Second, I don't buy that going beyond linear models is asking for statistical trouble. It just ain't so. People who overfit can (and actually do, all the time) stuff a ton of variables into a linear model and successfully overfit this way.

And the number of terms explode when you add non linearities.

5 independent variables with quadratic terms give you 21 values to play with (1 constant + 5 linear + 15 quadratic); it's much easier to justify conceptually "lets look at quadratic terms" than "lets add in 15 extra variables" even though the effect on degrees of freedom is the same.

Comment author:Lumifer
30 July 2014 04:46:43PM
2 points
[-]

And the number of terms explode when you add non linearities

No, they don't. You control the number of degrees of freedom in your models. If you don't, linear models won't help you much, and if you do linearity does not matter.

5 independent variables with quadratic terms give you 21 values to play with

I think you're confusing quadratic terms and interaction terms. It also seems that you're thinking of linear models solely as linear regressions. Do you consider, e.g. GLMs to be "linear" models? What about transformations of input variables, are they disallowed in your understanding of linear models?

I'm talking about practice, not theory. And most of the practical results that I've seen is that regression linear models are full of overfitting if they aren't linear. Even beyond human error, it seems that in many social science areas the data quality is poor enough that adding non-linearities can be seen, a priori, to be a bad thing to do.

Except of course if there is a firm reason to add a particular non-linearity to the problem.

I'm not familiar with the whole spectrum of models (regression models, beta distributions, some conjugate prior distributions, and some machine learning techniques is about all I know), so I can't confidently speak about the general case. But, extrapolating from what I've seen and known biases and incentives, I'm quite confident in predicting that generic models are much more likely to be overfitted than to have too few degrees of freedom.

Comment author:[deleted]
31 July 2014 09:20:28PM
2 points
[-]

How do I account for how many models I've tested? No, really, I don't know what that'd even be called in the statistics literature, and it seems like if a general technique for doing this were known the big data people would be all over it.

What we're doing at the FHI is acting like a machine learning problem: splitting the data into a training and a testing set, checking as much as we want on the training set, formulating the hypotheses, then testing them on the testing set.

Comment author:EHeller
07 August 2014 05:14:24PM
*
4 points
[-]

I see a lot of stepwise regression being used by non-statisticians, but I think statisticians themselves think its something of a joke. If you have more predictors than you can fit coefficients for, and want an understandable linear model you are better off with something like LASSO.

Comment author:[deleted]
07 August 2014 04:30:25PM
1 point
[+]
(0
children)

Comment author:[deleted]
07 August 2014 04:30:25PM
1 point
[-]

So it wasn't as clear with the previous link, but it seems to me that the nth step of this method doesn't condition on the fact that the last n-1 steps failed.

If you array the full might of statistics/machine learning/knowledge representation in AI/math/signal processing, and took the very best, I am very sure they could beat a linear model for a non-linear ground truth very easily. If so, maybe the right thing to do here is to emulate those people when doing data analysis, and not use the model we know to be wrong.

Comment author:Lumifer
28 July 2014 06:52:45PM
4 points
[-]

Because in many fields, linear models (even poor ones) are the best we're going to get, with more complex models losing to overfitting.

That's privileging a particular class of models just because they historically were easy to calculate.

If you're concerned about overfitting you need to be careful with how many parameters are you using, but that does not translate into an automatic advantage of a linear model over, say, a log one.

The article you linked to goes to pre-(personal)computer times when dealing with non-linear models was often just impractical.

Comment author:gwern
28 July 2014 06:52:44PM
2 points
[-]

Because in many fields, linear models (even poor ones) are the best we're going to get, with more complex models losing to overfitting.

I don't think that's true. What fields show optimal performance from linear models where better predictions can't be gotten from other techniques like decision trees or neural nets or ensembles of techniques?

Well there's Goldberg, Lewis R. "Five models of clinical judgment: An empirical comparison between linear and nonlinear representations of the human inference process." Organizational Behavior and Human Performance 6.4 (1971): 458-479.

The main thing is that these old papers seem to still be considered valid, see eg Shanteau, James. "How much information does an expert use? Is it relevant?." Acta Psychologica 81.1 (1992): 75-86.

Comment author:gwern
05 August 2014 08:38:53PM
*
6 points
[-]

(It would be nice if you would link fulltext instead of providing citations; if you don't have access to the fulltext, it's a bad idea to cite it, and if you do, you should provide it for other people who are trying to evaluate your claims and whether the paper is relevant or wrong.)

A third possibility is that incorrect methods were used to measure the amount of information in experts’ judgments; use of the “correct” measurement method might support the Information-Use Hypothesis. In the studies reported here, four techniques were used to measure information use: protocol analysis, multiple regression analysis, analysis of variance, and self-ratings by judges. Despite differences in measurement methods, comparable results were reported. Other methodological issues might be raised, but the studies seem varied enough to rule out any artifactual explanation.

These aren't very good methods for extracting the full measure of information.

So to summarize: reality isn't entirely linear, so nonlinear methods frequently excel with modern developments to regularize and avoid overfitting (we can see this in the low prevalence of linear methods in demanding AI tasks like image recognition, or more generally, competitions like Kaggle on all sorts of domains); to the extent that humans are good predictors and classifiers too of reality, their predictions/classifications will be better mimicked by nonlinear methods; research showing the contrary typically does not compare very good methods and much more recent research may do much better (for example, parole/recidivism predictions by parole boards may be bad and easily improved on by linear models, but does that mean algorithms can't do even better?), and to the extent linear methods succeed, it may reflect the lack of relevant data or inherent randomness of results for a particular cherrypicked task.

I tend to agree with you about models, once overfitting is sorted.

to the extent that humans are good predictors and classifiers too of reality, their predictions/classifications will be better mimicked by nonlinear methods

## Comments (90)

Best*6 points [-]Good post.

If the ellipse is very narrow, things are indeed well-modeled by a linear relationship, and the biggest Y coordinate for a point is likely to also have close to biggest X coordinate.

If the ellipse is not narrow, that could be for two reasons. Either the underlying truth is indeed linear, but your data is very noisy. Or the underlying truth is not linear, and you should not use a linear model. (Or both, naturally).

If the underlying truth is linear, but your data is very noisy, then what happens to the X coordinate of points with given Y values is mostly determined by the noise.

If the underlying truth is not linear, why should we expect sensible answers from a linear model?

Because in many fields, linear models (even poor ones) are the best we're going to get, with more complex models losing to overfitting.

http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=1979-30170-001

*4 points [-]I don't follow you. Overfitting happens when your model has too many parameters, relative to the amount of data you have. It is true that linear models may have few parameters compared to some non-linear models (for example linear regression models vs regression models with extra interaction parameters). But surely, we can have sparsely parameterized non-linear models as well.

All I am saying is that if things are surprising it is either due to "noise" (variance) or "getting the truth wrong" (bias). Or both.

I agree that "models we can quickly and easily use while under publish-or-perish pressure" is an important class of models in practice :). Moreover, linear models are often in this class, while a ton of very interesting non-linear models in stats are not, and thus are rarely used. It is a pity.

A technical difficulty with saying that overfitting happens when there are "too many parameters" is that the parameters may do arbitrarily complicated things. For example they may encode C functions, in which case a model with a single (infinite-precision) real parameter can fit anything very well! Functions that are linear in their parameters and inputs do not suffer from this problem; the number of parameters summarizes their overfitting capacity well. The same is not true of some nonlinear functions.

To avoid confusion it may be helpful to define overfitting more precisely. The gist of any reasonable definition of overfitting is: If I randomly perturb the desired outputs of my function, how well can I find new parameters to fit the new outputs? I can't do a good job of giving more detail than that in a short comment, but if you feel confused about overfitting, here's a good (and famous) article about frequentist learning theory by Vladimir Vapnik that may be useful:

http://web.mit.edu/6.962/www/www_spring_2001/emin/slt.pdf

*2 points [-]This is about "reasonable encoding" not "linearity," though. That is, linear functions of parameters encode reasonably, but not all reasonable encodings are linear. We can define a parameter to be precisely one bit of information, and then ask for the minimum of bits needed.

I don't understand why people are so hung up on linearity.

Sure, technically if Alice fits a small noisy data set as y(x) = a*x+b and Bob fits it as y(x) = c*Ai(d*x) (where Ai is the Airy function) they've used the same number of parameters, but that won't stop me from rolling my eyes at the latter unless he has a good first-principle reason to privilege the hypothesis.

The problem is more practical than theoretical (don't have the links to hand. but you can find some in my silos of expertise post). Statisticians do not adjust properly for extra degrees of freedom, so among some category of published models, the linear ones will be best. Also, it seems that linear models are very good for modelling human expertise - we might think we're complex, but we behave pretty linearly.

*2 points [-]"Statisticians" is a pretty large set.

I still don't understand your original "because." I am talking about modeling the truth, not modeling what humans do. If the truth is not linear and humans use a linear modeling algorithm, well then they aren't a very good role model are they?

[ edit: did not downvote. ]

Because human flaws creep in in the process of modelling as well. Taking non linear relationships into account (unless there is a causal reason to do so) is asking for statistical trouble unless you very carefully account for how many models you have tested and tried (which almost nobody does).

First, the structure of your model should be driven by the structure you're observing in your data. If you are observing nonlinearities, you'd better model nonlinearities.

Second, I don't buy that going beyond linear models is asking for statistical trouble. It just ain't so. People who overfit can (and actually do, all the time) stuff a ton of variables into a linear model and successfully overfit this way.

And the number of terms explode when you add non linearities.

5 independent variables with quadratic terms give you 21 values to play with (1 constant + 5 linear + 15 quadratic); it's much easier to justify conceptually "lets look at quadratic terms" than "lets add in 15 extra variables" even though the effect on degrees of freedom is the same.

No, they don't. You

controlthe number of degrees of freedom in your models. If you don't, linear models won't help you much, and if you do linearity does not matter.I think you're confusing quadratic terms and interaction terms. It also seems that you're thinking of linear models solely as linear regressions. Do you consider, e.g. GLMs to be "linear" models? What about transformations of input variables, are they disallowed in your understanding of linear models?

I'm talking about practice, not theory. And most of the practical results that I've seen is that regression linear models are full of overfitting if they aren't linear. Even beyond human error, it seems that in many social science areas the data quality is poor enough that adding non-linearities can be seen, a priori, to be a bad thing to do.

Except of course if there is a firm reason to add a particular non-linearity to the problem.

I'm not familiar with the whole spectrum of models (regression models, beta distributions, some conjugate prior distributions, and some machine learning techniques is about all I know), so I can't confidently speak about the general case. But, extrapolating from what I've seen and known biases and incentives, I'm quite confident in predicting that generic models are much more likely to be overfitted than to have too few degrees of freedom.

How

doI account for how many models I've tested? No, really, I don't know what that'd even be called in the statistics literature, and it seems like if a general technique for doing this were known the big data people would be all over it.What we're doing at the FHI is acting like a machine learning problem: splitting the data into a training and a testing set, checking as much as we want on the training set, formulating the hypotheses, then testing them on the testing set.

The Bayesian approach with multiple models seems to be exactly what we need. eg http://www.stat.washington.edu/raftery/Research/PDF/socmeth1995.pdf

Another approach seems to be stepwise regression: http://en.wikipedia.org/wiki/Stepwise_regression

*4 points [-]I see a lot of stepwise regression being used by non-statisticians, but I think statisticians themselves think its something of a joke. If you have more predictors than you can fit coefficients for, and want an understandable linear model you are better off with something like LASSO.

Edit: Don't just take my word for it, google found this blog post for me: http://andrewgelman.com/2014/06/02/hate-stepwise-regression/

I concur. Stepwise regression is a very crude technique.

I find it useful as an initial filter if I have to dig through a LOT of potential predictors, but you can't rely on it to produce a decent model.

So it wasn't as clear with the previous link, but it seems to me that the nth step of this method doesn't condition on the fact that the last n-1 steps failed.

*2 points [-]If you array the full might of statistics/machine learning/knowledge representation in AI/math/signal processing, and took the very best, I am very sure they could beat a linear model for a non-linear ground truth very easily. If so, maybe the right thing to do here is to emulate those people when doing data analysis, and not use the model we know to be wrong.

Proper Bayesianism will triumph! But not in the hands of everyone.

That's privileging a particular class of models just because they historically were easy to calculate.

If you're concerned about overfitting you need to be careful with how many parameters are you using, but that does not translate into an automatic advantage of a linear model over, say, a log one.

The article you linked to goes to pre-(personal)computer times when dealing with non-linear models was often just impractical.

I don't think that's true. What fields show optimal performance from linear models where better predictions can't be gotten from other techniques like decision trees or neural nets or ensembles of techniques?

Showing that crude linear models, with no form of regularization or priors, beats human clinical judgement, doesn't show your previous claim.

Modelling human clinical judgement is best done with linear models, for instance.

Bestdone? Better than, say, decision trees or expert systems or Bayesian belief networks? Citation needed.Goldberg, Lewis R. "Simple models or simple processes? Some research on clinical judgments." American Psychologist 23.7 (1968): 483.

1968? Seriously?

Well there's Goldberg, Lewis R. "Five models of clinical judgment: An empirical comparison between linear and nonlinear representations of the human inference process." Organizational Behavior and Human Performance 6.4 (1971): 458-479.

The main thing is that these old papers seem to still be considered valid, see eg Shanteau, James. "How much information does an expert use? Is it relevant?." Acta Psychologica 81.1 (1992): 75-86.

*6 points [-](It would be nice if you would link fulltext instead of providing citations; if you don't have access to the fulltext, it's a bad idea to cite it, and if you do, you should provide it for other people who are trying to evaluate your claims and whether the paper is relevant or wrong.)

I've put up the first paper at https://dl.dropboxusercontent.com/u/85192141/1971-goldberg.pdf / https://pdf.yt/d/Ux7RZXbo0n374dUU I don't think this is particularly relevant: it only shows that 2 very specific equations (pg4, #3 & #4) did not outperform the linear model on a particular dataset. Too bad for Einhorn 1971.

Your second paper doesn't support the claims:

These aren't very good methods for extracting the full measure of information.

So to summarize: reality isn't entirely linear, so nonlinear methods frequently excel with modern developments to regularize and avoid overfitting (we can see this in the low prevalence of linear methods in demanding AI tasks like image recognition, or more generally, competitions like Kaggle on all sorts of domains); to the extent that humans are good predictors and classifiers too of reality, their predictions/classifications will be better mimicked by nonlinear methods; research showing the contrary typically does not compare very good methods and much more recent research may do much better (for example, parole/recidivism predictions by parole boards may be bad and easily improved on by linear models, but does that mean algorithms can't do even better?), and to the extent linear methods succeed, it may reflect the lack of relevant data or inherent randomness of results for a particular cherrypicked task.

To show your original claim ("in many fields, linear models (even poor ones) are the best we're going to get, with more complex models losing to overfitting"), I would want to see linear models steadily beat all comers, from random forests to deep neural networks to ensembles of all of the above, on a wide variety of large datasets. I don't think you can show that.

I tend to agree with you about models, once overfitting is sorted.

This I've still seen no evidence for.