Learning Data Mining with Python

Extracting Features with Transformers
Principal Component Analysis
In some datasets, features heavily correlate with each other. For example, the speed
and the fuel consumption would be heavily correlated in a go-kart with a single gear.
While it can be useful to find these correlations for some applications, data mining
algorithms typically do not need the redundant information.
The ads dataset has heavily correlated features, as many of the keywords are
repeated across the alt text and caption.
The Principal Component Analysis (PCA) aims to find combinations of
features that describe the dataset in less information. It aims to discover principal
components, which are features that do not correlate with each other and explain the
information—specifically the variance—of the dataset. What this means is that we
can often capture most of the information in a dataset in fewer features.
We apply PCA just like any other transformer. It has one key parameter, which is the
number of components to find. By default, it will result in as many features as you
have in the original dataset. However, these principal components are ranked—the
first feature explains the largest amount of the variance in the dataset, the second a
little less, and so on. Therefore, finding just the first few features is often enough to
explain much of the dataset. Let's look at the code:
from sklearn.decomposition import PCA
pca = PCA(n_components=5)
Xd = pca.fit_transform(X)
The resulting matrix, Xd, has just five features. However, let's look at the amount of
variance that is explained by each of these features:
np.set_printoptions(precision=3, suppress=True)
pca.explained_variance_ratio_
The result, array([ 0.854, 0.145, 0.001, 0. , 0. ]), shows
us that the first feature accounts for 85.4 percent of the variance in the dataset,
the second accounts for 14.5 percent, and so on. By the fourth feature, less than
one-tenth of a percent of the variance is contained in the feature. The other 1,553
features explain even less.
[ 96 ]