anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Murphy and Pazzani (1994) reported a set of experiments in which all the possible
consistent decision trees were produced and showed that, for several tasks,
the smallest consistent decision tree had higher predictive error than slightly
larger trees. However, when the authors compared the likelihood of better generalization
for smaller vs. more complex trees, they concluded that simpler hypotheses
should be preferred when no further knowledge of the target concept
is available. The small number of training examples relative to the size of the
tree that perfectly describes the concept might explain why, in these cases, the
smallest tree did not generalize best. Another reason could be that only small
spaces of decision trees were explored. To verify these explanations, we use a similar
experimental setup where the datasets have larger training sets and attribute
vectors of higher dimensionality. Because the number of all possible consistent
trees is huge, we use a Random Tree Generator (RTG) to sample the space of
trees obtainable by TDIDT algorithms. RTG builds a tree top-down and chooses
the splitting attribute at random.
We report the results for three datasets: XOR-5, Tic-tac-toe, and Zoo (See
Appendix A for detailed descriptions of these datasets). For each dataset, the
examples were partitioned into a training set (90%) and a testing set (10%), and
RTG was used to generate a sample of 10 million consistent decision trees. The
behavior in the three datasets is similar: the accuracy monotonically decreases
with the increase in the size of the trees (number of leaves), confirming the utility
of Occam’s Razor. In Appendix B we report further experiments with a variety
of datasets, which indicate that the inverse correlation between size and accuracy
is statistically significant (Esmeir & Markovitch, 2007c).
It is important to note that smaller trees have several advantages aside from
their probable greater accuracy, such as greater statistical evidence at the leaves,
better comprehensibility, lower storage costs, and faster classification (in terms
of total attribute evaluations).
Motivated by the above discussion, our goal is to find the smallest consistent
tree. In TDIDT, the learner has to choose a split at each node, given a set of
examples E that reach the node and a set of unused attributes A. For each attribute
a, let Tmin(a, A, E) be the smallest tree rooted at a that is consistent with
E and uses attributes from A − {a} for the internal splits. Given two candidate
attributes a1 and a2, we obviously prefer the attribute whose associated Tmin(a)
is the smaller. For a set of attributes A, we define � A to be the set of attributes
whose associated Tmin(a) is the smaller. Formally, � A = arg mina∈A Tmin (a). We
say that a splitting attribute a is optimal if a ∈ � A. Observe that if the learner
makes an optimal decision at each node, then the final tree is necessarily globally
optimal.
23