Building Machine Learning Systems with Python - Richert, Coelho

Chapter 7
So, we can see that the data lies between -7.9 and -0.5. Now that we have an estimate
data, we can check what happens when we use OLS to predict. Note that we can use
exactly the same classes and methods as before:
from sklearn.linear_model import LinearRegression
lr = LinearRegression(fit_intercept=True)
lr.fit(data,target)
p = np.array(map(lr.predict, data))
p = p.ravel() # p is a (1,16087) array, we want to flatten it
e = p-target # e is 'error': difference of prediction and reality
total_sq_error = np.sum(e*e)
rmse_train = np.sqrt(total_sq_error/len(p))
print(rmse_train)
The error is not exactly zero because of the rounding error, but it is very close:
0.0025 (much smaller than the standard deviation of the target, which is the natural
comparison value).
When we use cross-validation (the code is very similar to what we used before in the
Boston example), we get something very different: 0.78. Remember that the standard
deviation of the data is only 0.6. This means that if we always "predict" the mean
value of -3.5, we have a root mean square error of 0.6! So, with OLS, in training, the
error is insignificant. When generalizing, it is very large and the prediction is actually
harmful: we would have done better (in terms of root mean square error) by simply
predicting the mean value every time!
Training and generalization error
When the number of features is greater than the number of examples,
you always get zero training error with OLS, but this is rarely a sign
that your model will do well in terms of generalization. In fact, you
may get zero training error and have a completely useless model.
One solution, naturally, is to use regularization to counteract the overfitting. We can
try the same cross-validation loop with an elastic net learner, having set the penalty
parameter to 1. Now, we get 0.4 RMSE, which is better than just "predicting the
mean". In a real-life problem, it is hard to know when we have done all we can as
perfect prediction is almost always impossible.
[ 157 ]