anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
wrong greedy decisions. When there are sufficient examples, wrong splits do
not significantly diminish accuracy, but result in less comprehensible trees and
costlier classification. When there are fewer examples the problem is intensified
because the accuracy is adversely affected. In our proposed anytime approach,
this problem is tackled by allotting additional time for reaching trees unexplored
by greedy algorithms with pruning. Therefore, we view pruning as orthogonal
to lookahead and suggest considering it as an additional phase in order to avoid
overfitting.
Pruning can be applied in two different ways. First, it can be applied as a
second phase after each LSID3 final tree has been built. We denote this extension
by LSID3-p (and BLSID3-p for binary splits). Second, it is worthwhile to examine
pruning of the lookahead trees. For this purpose, a post-pruning phase will be
applied to the trees that were generated by SID3 to form the lookahead sample,
and the size of the pruned trees will be considered. The expected contribution
of pruning in this case is questionable since it has a similar effect on the entire
lookahead sample.
Previous comparative studies did not find a single pruning method that is
generally the best and conclude that different pruning techniques behave similarly
(Esposito, Malerba, & Semeraro, 1997; Oates & Jensen, 1997). Therefore, in our
experiments we adopt the same pruning method as in C4.5, namely error-based
pruning (EPB), and examine post-pruning of the final trees as well as pruning of
the lookahead trees.
Figure 3.10 describes the spaces of decision trees obtainable by LSID3. Let
A be a set of attributes and E be a set of examples. The space of decision trees
over A can be partitioned into two: the space of trees consistent with E and the
space of trees inconsistent with E. The space of consistent trees includes as a
subspace the trees inducible by TDIDT. 7 The trees that ID3 can produce are a
subspace of the TDIDT space. Pruning extends the space of ID3 trees to the
inconsistent space. LSID3 can reach trees that are obtainable by TDIDT but not
by the greedy methods and their extensions. LSID3-p expands the search space
to trees that do not fit the training data perfectly, so that noisy training sets can
be handled.
3.5.4 Mapping Contract Time to Sample Size
LSID3 is parameterized by r, the number of times SID3 is invoked for each
candidate. In real-life applications, however, the user supplies the system with
7 TDIDT is a strict subspace because it cannot induce trees that: (1) contain a subtree with
all leaves marked with the same class, and (2) include an internal node that has no associated
training examples.
33