Building Machine Learning Systems with Python - Richert, Coelho

Dimensionality Reduction
Asking the model about the features
using wrappers
While filters can tremendously help in getting rid of useless features, they can
go only so far. After all the filtering, there might still be some features that are
independent among themselves and show some degree of dependence with the
result variable, but yet they are totally useless from the model's point of view. Just
think of the following data, which describes the XOR function. Individually, neither
A nor B would show any signs of dependence on Y, whereas together they clearly do:
A B Y
0 0 0
0 1 1
1 0 1
1 1 0
So why not ask the model itself to give its vote on the individual features?
This is what wrappers do, as we can see in the following process chart diagram:
y
Current
features,
initialized with
all features
x1, x2, ..., xN
Train model
with y and check
the importance
of individual
features
Importance of
individual
features
Feature set too big
No
Resulting
features
x2, x10, x14
Yes
Drop features
that are
unimportant
Here we have pushed the calculation of feature importance to the model training
process. Unfortunately (but understandably), feature importance is not determined
as a binary but as a ranking value. So we still have to specify where to make the
cut – what part of the features are we willing to take and what part do we want
to drop?
[ 230 ]