Learning Data Mining with Python

To compensate for this, we could create many decision trees and then ask each to
predict the class value. We could take a majority vote and use that answer as our
overall prediction. Random forests work on this principle.
Chapter 3
There are two problems with the aforementioned procedure. The first problem is
that building decision trees is largely deterministic—using the same input will result
in the same output each time. We only have one training dataset, which means our
input (and therefore the output) will be the same if we try build multiple trees. We
can address this by choosing a random subsample of our dataset, effectively creating
new training sets. This process is called bagging.
The second problem is that the features that are used for the first few decision nodes
in our tree will be quite good. Even if we choose random subsamples of our training
data, it is still quite possible that the decision trees built will be largely the same. To
compensate for this, we also choose a random subset of the features to perform our
data splits on.
Then, we have randomly built trees using randomly chosen samples, using (nearly)
randomly chosen features. This is a Random Forest and, perhaps unintuitively,
this algorithm is very effective on many datasets.
How do ensembles work?
The randomness inherent in Random forests may make it seem like we are leaving
the results of the algorithm up to chance. However, we apply the benefits of
averaging to nearly randomly built decision trees, resulting in an algorithm that
reduces the variance of the result.
Variance is the error introduced by variations in the training dataset on the
algorithm. Algorithms with a high variance (such as decision trees) can be greatly
affected by variations to the training dataset. This results in models that have the
problem of overfitting.
In contrast, bias is the error introduced by assumptions in the
algorithm rather than anything to do with the dataset, that is, if we
had an algorithm that presumed that all features would be normally
distributed then our algorithm may have a high error if the features
were not. Negative impacts from bias can be reduced by analyzing the
data to see if the classifier's data model matches that of the actual data.
[ 55 ]