Learning Data Mining with Python

Predicting Sports Winners with Decision Trees
There are many algorithms for creating decision trees. Many of these algorithms
are iterative. They start at the base node and decide the best feature to use for the
first decision, then go to each node and choose the next best feature, and so on. This
process is stopped at a certain point, when it is decided that nothing more can be
gained from extending the tree further.
The scikit-learn package implements the CART (Classification and Regression
Trees) algorithm as its default decision tree class, which can use both categorical and
continuous features.
Parameters in decision trees
One of the most important features for a decision tree is the stopping criterion.
As a tree is built, the final few decisions can often be somewhat arbitrary and rely
on only a small number of samples to make their decision. Using such specific nodes
can results in trees that significantly overfit the training data. Instead, a stopping
criterion can be used to ensure that the decision tree does not reach this exactness.
Instead of using a stopping criterion, the tree could be created in full and then
trimmed. This trimming process removes nodes that do not provide much
information to the overall process. This is known as pruning.
The decision tree implementation in scikit-learn provides a method to stop the
building of a tree using the following options:
• min_samples_split: This specifies how many samples are needed in order
to create a new node in the decision tree
• min_samples_leaf: This specifies how many samples must be resulting
from a node for it to stay
The first dictates whether a decision node will be created, while the second dictates
whether a decision node will be kept.
Another parameter for decision tress is the criterion for creating a decision.
Gini impurity and Information gain are two popular ones:
• Gini impurity: This is a measure of how often a decision node would
incorrectly predict a sample's class
• Information gain: This uses information-theory-based entropy to indicate
how much extra information is gained by the decision node
[ 48 ]