Learning Data Mining with Python

Extracting Features with Transformers
Feature selection
We will often have a large number of features to choose from, but we wish to select
only a small subset. There are many possible reasons for this:
• Reducing complexity: Many data mining algorithms need more time and
resources with increase in the number of features. Reducing the number
of features is a great way to make an algorithm run faster or with fewer
resources.
• Reducing noise: Adding extra features doesn't always lead to better
performance. Extra features may confuse the algorithm, finding correlations
and patterns that don’t have meaning (this is common in smaller datasets).
Choosing only the appropriate features is a good way to reduce the chance of
random correlations that have no real meaning.
• Creating readable models: While many data mining algorithms will happily
compute an answer for models with thousands of features, the results may be
difficult to interpret for a human. In these cases, it may be worth using fewer
features and creating a model that a human can understand.
Some classification algorithms can handle data with issues such as these. Getting
the data right and getting the features to effectively describe the dataset you are
modeling can still assist algorithms.
There are some basic tests we can perform, such as ensuring that the features are at
least different. If a feature's values are all same, it can't give us extra information to
perform our data mining.
The VarianceThreshold transformer in scikit-learn, for instance, will remove any
feature that doesn't have at least a minimum level of variance in the values. To show
how this works, we first create a simple matrix using NumPy:
import numpy as np
X = np.arange(30).reshape((10, 3))
The result is the numbers zero to 29, in three columns and 10 rows. This represents a
synthetic dataset with 10 samples and three features:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[ 88 ]