anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
4.3.2 Choosing a Split when Attributes are Numeric
The selection procedure as formalized in Figure 4.2 must be modified slightly
when an attribute is numeric: rather than iterating over the values the attribute
can take, we first pick r tests (split points) with the highest information gain and
then invoke EG2 once for each point. This guarantees that numeric and nominal
attributes get the same resources. Chickering, Meek, and Rounthwaite (2001)
introduced several techniques for generating a small number of candidate split
points dynamically with little overhead. In the future we intend to apply these
techniques to select the r points. Alternatively, we could apply the Monte-carlo
method described in Section 3.5.1, which is more computationally expensive due
to the need to sort the points.
4.4 Cost-sensitive Pruning
Pruning plays an important role in decision tree induction. In cost-insensitive
environments, the main goal of pruning is to simplify the tree in order to avoid
overfitting the training data. A subtree is pruned if the resulting tree is expected
to yield a lower error.
When test costs are taken into account, pruning has another important role:
reducing test costs in a tree. Keeping a subtree is worthwhile only if its expected
reduction in misclassification costs is larger than the cost of the tests in that
subtree. If the misclassification cost is zero, it makes no sense to keep any split
in the tree. If, on the other hand, the misclassification cost is much larger than
the test costs, we would expect similar behavior to the cost-insensitive setup.
To handle this challenge, we propose a novel approach for cost-sensitive pruning.
As in error-based pruning (Quinlan, 1993), we scan the tree bottom-up.
Then we compare the expected total cost of each subtree to that of a leaf. If a
leaf is expected to perform better, the subtree is pruned.
The cost of a subtree is estimated as described in Section 4.2. Formally, let
E be the set of training examples that reach a subtree T, and let m be the size
of E. Assume that s examples in E do not belong to the default class. 2 Let L be
the set of leaves in T. We prune T into a leaf if:
1
m · EE(m, s, cf) · mc ≤ tcost(T, � E) + �
l∈L
mcost(l). �
The above assumes a uniform misclassification cost mc. In the case of nonuniform
penalties, we multiply the expected error by the average misclassification cost.
2 If misclassification costs are uniform, the default class is the majority class. Otherwise, it
is the class that minimizes the misclassification cost in the node.
78