From Algorithms to Z-Scores - matloff - University of California, Davis

308 CHAPTER 15. RELATIONS AMONG VARIABLES: LINEAR REGRESSION
Here I use the term model selection to mean which predictor variables (including powers and
interactions) we will use. If we have data on many predictors, we almost certainly will not be able
to use them all, for the following reason:
15.17.1 The Overfitting Problem in Regression
Recall in Section 15.10) we mentioned that we could add polynomial terms to a regression model.
But you can see that if we carry this notion to its extreme, we get absurd results. If we fit a
polynomial of degree 99 to our 100 points, we can make our fitted curve exactly pass through every
point! This clearly would give us a meaningless, useless curve. We are simply fitting the noise.
Recall that we analyzed this problem in Section 14.1.4 in our chapter on modeling. There we noted
an absolutely fundamental principle in statistics:
In choosing between a simpler model and a more complex one, the latter is more accurate
only if either
• we have enough data to support it, or
• the complex model is sufficiently different from the simpler one
This is extremely important in regression analysis, because we often have so many
variables we can use, thus often can make highly complex models.
In the regression context, the phrase “we have enough data to support the model” means (in the
parametric model case) we have enough data so that the confidence intervals for the βi will be
reasonably narrow. For fixed n, the more complex the model, the wider the resulting confidence
intervals will tend to be.
If we use too many predictor variables, 11 , our data is “diluted,” by being “shared” by so many
βi. As a result, V ar( βi) will be large, with big implications: Whether our goal is Prediction or
Understanding, our estimates will be so poor that neither goal is achieved.
On the other hand, if some predictor variable is really important (i.e. its βi is far from 0), then it
may pay to include it, even though the confidence intervals might get somewhat wider.
The questions raised in turn by the above considerations, i.e. How much data is enough data?,
and How different from 0 is “quite different”?, are addressed below in Section 15.17.3.
A detailed mathematical example of overfitting in regression is presented in my paper A Careful
Look at the Use of Statistical Methodology in Data Mining (book chapter), by N. Matloff, in
11 In the ALOHA example above, b, b 2 , b 3 and b 4 are separate predictors, even though they are of course correlated.