Building Machine Learning Systems with Python - Richert, Coelho

Dimensionality Reduction
Sketching PCA
PCA involves a lot of linear algebra, which we do not want to go into. Nevertheless,
the basic algorithm can be easily described with the help of the following steps:
1. Center the data by subtracting the mean from it.
2. Calculate the covariance matrix.
3. Calculate the eigenvectors of the covariance matrix.
If we start with features, the algorithm will again return a transformed feature
space with dimensions – we gained nothing so far. The nice thing about this
algorithm, however, is that the eigenvalues indicate how much of the variance is
described by the corresponding eigenvector.
Let us assume we start with features, and we know that our model does
not work well with more than 20 features. Then we simply pick the 20 eigenvectors
having the highest eigenvalues.
Applying PCA
Let us consider the following artificial dataset, which is visualized in the left plot
as follows:
>>> x1 = np.arange(0, 10, .2)
>>> x2 = x1+np.random.normal(loc=0, scale=1, size=len(x1))
>>> X = np.c_[(x1, x2)]
>>> good = (x1>5) | (x2>5) # some arbitrary classes
>>> bad = ~good # to make the example look good
[ 234 ]