Building Machine Learning Systems with Python - Richert, Coelho

Chapter 11
Feature extraction
At some point, after we have removed the redundant features and dropped the
irrelevant ones, we often still find that we have too many features. No matter what
learning method we use, they all perform badly, and given the huge feature space,
we understand that they actually cannot do better. We realize that we have to cut
living flesh and that we have to get rid of features that all common sense tells us
are valuable. Another situation when we need to reduce the dimensions, and when
feature selection does not help much, is when we want to visualize data. Then, we
need to have at most three dimensions at the end to provide any meaningful graph.
Enter the feature extraction methods. They restructure the feature space to make it
more accessible to the model, or simply cut down the dimensions to two or three so
that we can show dependencies visually.
Again, we can distinguish between feature extraction methods as being linear
or non-linear ones. And as before, in the feature selection section, we will
present one method for each type, principal component analysis for linear and
multidimensional scaling for the non-linear version. Although they are widely
known and used, they are only representatives for many more interesting and
powerful feature extraction methods.
About principal component analysis (PCA)
Principal component analysis is often the first thing to try out if you want to cut
down the number of features and do not know what feature extraction method to
use. PCA is limited as it is a linear method, but chances are that it already goes far
enough for your model to learn well enough. Add to that the strong mathematical
properties it offers, the speed at which it finds the transformed feature space, and
its ability to transform between the original and transformed features later, we
can almost guarantee that it will also become one of your frequently used machine
learning tools.
Summarizing it, given the original feature space, PCA finds a linear projection of it
into a lower dimensional space that has the following properties:
• The conserved variance is maximized
• The final reconstruction error (when trying to go back from transformed
features to original ones) is minimized
As PCA simply transforms the input data, it can be applied both to classification
and regression problems. In this section, we will use a classification task to discuss
the method.
[ 233 ]