Building Machine Learning Systems with Python - Richert, Coelho

Chapter 11
Coming back to Scikit-learn, we find various excellent wrapper classes in the sklearn.
feature_selection package. A real workhorse in this field is RFE, which stands for
recursive feature elimination. It takes an estimator and the desired number of features
to keep as parameters and then trains the estimator with various feature sets as long
as it has found a subset of the features that are small enough. The RFE instance itself
pretends to be like an estimator, thereby, wrapping the provided estimator.
In the following example, we create an artificial classification problem of 100 samples
using the convenient make_classification() function of datasets. It lets us
specify the creation of 10 features, out of which only three are really valuable to solve
the classification problem:
>>> from sklearn.feature_selection import RFE
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.datasets import make_classification
>>> X,y = make_classification(n_samples=100, n_features=10, n_
informative=3, random_state=0)
>>> clf = LogisticRegression()
>>> clf.fit(X, y)
>>> selector = RFE(clf, n_features_to_select=3)
>>> selector = selector.fit(X, y)
>>> print(selector.support_)
[False True False True False False False False True False]
>>> print(selector.ranking_)
[4 1 3 1 8 5 7 6 1 2]
The problem in real-world scenarios is, of course, how can we know the right value
for n_features_to_select? Truth is, we can't. But most of the time, we can use a
sample of the data and play with it using different settings to quickly get a feeling for
the right ballpark.
The good thing is that we don't have to be that exact when using wrappers.
Let's try different values for n_features_to_select to see how support_ and
ranking_ change:
n_
features_
to_select support_ ranking_
1
[False False False True False False False False
False False] [ 6 3 5 1 10 7 9 8 2 4]
2
[False False False True False False False False
True False] [5 2 4 1 9 6 8 7 1 3]
3
[False True False True False False False False
True False] [4 1 3 1 8 5 7 6 1 2]
[ 231 ]