anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
node if resources allow. These would make challenging and interesting directions
for future research.
Nijssen and Fromont (2007) presented DL8, an exact algorithm for finding a
decision tree that optimizes a ranking function under size, depth, accuracy and
leaf constraints. The key idea behind DL8 is that constraints on decision trees
are simulated as constraints on itemsets. They show that optimal decision trees
can be extracted from lattices of itemsets in linear time. The applicability of
DL8, however, is limited by two factors: the number of itemsets that need to be
stored, and the time that it takes to compute these itemsets. In some cases, the
number of frequent itemsets is so large that it is impossible to compute or store
them within a reasonable amount of time or space.
Several researchers investigated the induction of shallow decision trees. Holte
(1993) reported an empirical study of the accuracy of rules that classify examples
on the basis of a single test (1R). Holte concluded that, on most real world
datasets, multilevel decision trees do not perform significantly better than onelevel
classification rules. Elomaa (1994), however, questioned the validity of these
conclusions. Elomaa argued that the small difference in accuracy between 1R and
C4.5 is still significant, and that the conclusions may have been the results of the
use of unrepresentative databases.
Auer, Holte, and Maass (1995) presented a novel algorithm, called T2, for
agnostic PAC-learning with decision trees of at most 2-levels. The computation
time of T2 is almost linear in the size of the training set. When tested empirically
on several datasets, T2 was shown to produce substantially simpler decision trees
with little or no loss in predictive power. Since one can prove that T2 is an
angostic PAC-learning algorithm, it is guaranteed to produce close to optimal 2level
decision trees given sufficiently large training data. A generalization to depth
d, however, would require computation time that is exponential in d. Dobkin,
Gunopulos, and Kasif (1996) described several algorithms and theoretical results
for learning optimal consistent decision trees of bounded depth. These methods,
nevertheless, are practical only when the tree depth is tiny.
Much research effort has also been invested in the theoretical aspects of decision
tree learning. A standard theoretical approach is to prove a bound on generalization
error as a function of the training error and the concept size (McAllester,
1998). Then, a concept optimizing the tradeoff between training error and concept
size, as expressed in the bound, is selected. These bounds depend on the size
of the training sample, but not on the sample itself. To improve them, Mansour
and McAllester (2000) constructed bounds that depend both on the structure of
the model and on the actual examples that form the sample.
Kushilevitz and Mansour (1993) presented a polynomial time algorithm for
learning decision trees with respect to the uniform distribution, using membership
queries. The considered decision tree model is an extension of the traditional
126