1FfUrl0

Dimensionality Reduction
n_
features_
to_select support_ ranking_
4
[False True False True False False False False
True True] [3 1 2 1 7 4 6 5 1 1]
5
[False True True True False False False False
True True] [2 1 1 1 6 3 5 4 1 1]
6
[ True True True True False False False False
True True] [1 1 1 1 5 2 4 3 1 1]
7
[ True True True True False True False False
True True] [1 1 1 1 4 1 3 2 1 1]
8
[ True True True True False True False True
True True] [1 1 1 1 3 1 2 1 1 1]
9
[ True True True True False True True True
True True] [1 1 1 1 2 1 1 1 1 1]
10
[ True True True True True True True True
True True] [1 1 1 1 1 1 1 1 1 1]
We see that the result is very stable. Features that have been used when requesting
smaller feature sets keep on getting selected when letting more features in. Finally
we rely on our train/test set splitting to warn us when we go in the wrong direction.
Other feature selection methods
There are several other feature selection methods that you will discover while
reading through machine learning literature. Some don't even look like feature
selection methods as they are embedded into the learning process (not to be confused
with the previously mentioned wrappers). Decision trees, for instance, have a feature
selection mechanism implanted deep in their core. Other learning methods employ
some kind of regularization that punishes model complexity, thus driving the
learning process towards good performing models that are still "simple". They do
this by decreasing the less impactful features' importance to zero and then dropping
them (L1-regularization).
So watch out! Often, the power of machine learning methods has to be attributed to
their implanted feature selection method to a great degree.
[ 232 ]