Building Machine Learning Systems with Python - Richert, Coelho

Regression – Recommendations
The LinearRegression class implements OLS regression as follows:
lr = LinearRegression(fit_intercept=True)
We set the fit_intercept parameter to True in order to add a bias term. This
is exactly what we had done before, but in a more convenient interface:
lr.fit(x,y)
p = map(lr.predict, x)
Learning and prediction are performed for classification as follows:
e = p-y
total_error = np.sum(e*e) # sum of squares
rmse_train = np.sqrt(total_error/len(p))
print('RMSE on training: {}'.format(rmse_train))
We have used a different procedure to compute the root mean square error on the
training data. Of course, the result is the same as we had before: 4.6 (it is always
good to have these sanity checks to make sure we are doing things correctly).
Now, we will use the KFold class to build a 10-fold cross-validation loop and test
the generalization ability of linear regression:
from sklearn.cross_validation import Kfold
kf = KFold(len(x), n_folds=10)
err = 0
for train,test in kf:
lr.fit(x[train],y[train])
p = map(lr.predict, x[test])
e = p-y[test]
err += np.sum(e*e)
rmse_10cv = np.sqrt(err/len(x))
print('RMSE on 10-fold CV: {}'.format(rmse_10cv))
With cross-validation, we obtain a more conservative estimate (that is, the error is
greater): 5.6. As in the case of classification, this is a better estimate of how well we
could generalize to predict prices.
Ordinary least squares is fast at learning time and returns a simple model,
which is fast at prediction time. For these reasons, it should often be the first
model that you use in a regression problem. However, we are now going to see
more advanced methods.
[ 152 ]