Learning Data Mining with Python

Chapter 5
The downside to transforming data with PCA is that these features are often complex
combinations of the other features. For example, the first feature of the preceding
code starts with [-0.092, -0.995, -0.024], that is, multiply the first feature in the
original dataset by -0.092, the second by -0.995, the third by -0.024. This feature has
1,558 values of this form, one for each of the original datasets (although many are
zeros). Such features are indistinguishable by humans and it is hard to glean much
relevant information from without a lot of experience working with them.
Using PCA can result in models that not only approximate the original dataset,
but can also improve the performance in classification tasks:
clf = DecisionTreeClassifier(random_state=14)
scores_reduced = cross_val_score(clf, Xd, y, scoring='accuracy')
The resulting score is 0.9356, which is (slightly) higher than our original score using
all of the original features. PCA won't always give a benefit like this, but it does more
often than not.
We are using PCA here to reduce the number of features in our dataset.
As a general rule, you shouldn't use it to reduce overfitting in your
data mining experiments. The reason for this is that PCA doesn't take
classes into account. A better solution is to use regularization. An
introduction, with code, is available at http://blog.datadive.
net/selecting-good-features-part-ii-linear-modelsand-regularization/.
Another advantage is that PCA allows you to plot datasets that you otherwise
couldn't easily visualize. For example, we can plot the first two features returned
by PCA.
First, we tell IPython to display plots inline and import pyplot:
%matplotlib inline
from matplotlib import pyplot as plt
Next, we get all of the distinct classes in our dataset (there are only two: is ad or
not ad):
classes = set(y)
We also assign colors to each of these classes:
colors = ['red', 'green']
We use zip to iterate over both lists at the same time:
for cur_class, color in zip(classes, colors):
[ 97 ]