1FfUrl0

Regression – Recommendations
Ridge, Lasso, and Elastic nets
These penalized models often go by rather interesting names. The
L1 penalized model is often called the Lasso, while an L2 penalized
model is known as Ridge regression. Of course, we can combine the
two and we obtain an Elastic net model.
Both the Lasso and the Ridge result in smaller coefficients than unpenalized regression.
However, the Lasso has the additional property that it results in more coefficients
being set to zero! This means that the final model does not even use some of its input
features, the model is sparse. This is often a very desirable property as the model
performs both feature selection and regression in a single step.
You will notice that whenever we add a penalty, we also add a weight λ, which
governs how much penalization we want. When λ is close to zero, we are very
close to OLS (in fact, if you set λ to zero, you are just performing OLS), and when
λ is large, we have a model which is very different from the OLS one.
The Ridge model is older as the Lasso is hard to compute manually. However, with
modern computers, we can use the Lasso as easily as Ridge, or even combine them
to form Elastic nets. An Elastic net has two penalties, one for the absolute value and
another for the squares.
Using Lasso or Elastic nets in scikit-learn
Let us adapt the preceding example to use elastic nets. Using scikit-learn, it is very
easy to swap in the Elastic net regressor for the least squares one that we had before:
from sklearn.linear_model import ElasticNet
en = ElasticNet(fit_intercept=True, alpha=0.5)
Now we use en whereas before we had used lr. This is the only change that is
needed. The results are exactly what we would have expected. The training error
increases to 5.0 (which was 4.6 before), but the cross-validation error decreases to
5.4 (which was 5.6 before). We have a larger error on the training data, but we gain
better generalization. We could have tried an L1 penalty using the Lasso class or L2
using the Ridge class with the same code.
The next plot shows what happens when we switch from unpenalized regression
(shown as a dotted line) to a Lasso regression, which is closer to a flat line. The
benefits of a Lasso regression are, however, more apparent when we have many
input variables and we consider this setting next:
[ 154 ]