Building Machine Learning Systems with Python - Richert, Coelho

Chapter 11
Detecting redundant features using filters
Filters try to clean up the feature forest independent of any machine learning
method used later. They rely on statistical methods to find out which of the features
are redundant (in which case, we need to keep only one per redundant feature
group) or irrelevant. In general, the filter works as depicted in the workflow
shown in the following diagram:
y
All features
x1, x2, ..., xN
Select features
that are not
redundant
Some features
x2, x7, ..., xM
Select features
that are not
irrelevant
Resulting
features
x2, x10, x14
Correlation
Using correlation, we can easily see linear relationships between pairs of features,
which are relationships that can be modeled using a straight line. In the graphs
shown in the following screenshot, we can see different degrees of correlation
together with a potential linear dependency plotted as a red dashed line (a fitted
one-dimensional polynomial). The correlation coefficient
at the top of
the individual graphs is calculated using the common Pearson correlation coefficient
(the Pearson value) by means of the pearsonr() function of scipy.stat.
Given two equal-sized data series, it returns a tuple of the correlation coefficient
values and the p-value, which is the probability that these data series are being
generated by an uncorrelated system. In other words, the higher the p-value, the
less we should trust the correlation coefficient:
>> from import scipy.stats import pearsonr
>> pearsonr([1,2,3], [1,2,3.1])
>> (0.99962228516121843, 0.017498096813278487)
>> pearsonr([1,2,3], [1,20,6])
>> (0.25383654128340477, 0.83661493668227405)
In the first case, we have a clear indication that both series are correlated. In the
second one, we still clearly have a non-zero value.
[ 223 ]