anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
simpler XOR and multiplexer problems, its limited lookahead is not sufficient
for learning complex concepts such as XOR-10: DMTI achieved an accuracy of
50%. IIDT and LSID3, by producing larger samples, overcame this problem and
reached high accuracies.
Kim and Loh (2001) introduced CRUISE, a bias-free decision tree learner
that attempts to produce more compact trees by (1) using multiway splits—
one subnode for each class, and (2) examining pair-wise interactions among the
variables. CRUISE is able to learn XOR-2 and Chess-board (numeric XOR-
2) concepts. Much like ID3-k with k = 2, it cannot recognize more complex
interactions.
Bennett (1994) presented GTO, a non-greedy approach for repairing multivariate
decision trees. GTO requires as input an initial tree. The algorithm
retains the structure of the tree but attempts to simultaneously improve all the
multivariate decisions of the tree using iterative linear programming. GTO and
IIDT both use a non-greedy approach to improve a decision tree. The advantage
of GTO is its use of a well-established numerical method for optimization. Its
disadvantages are its inability to modify the initial structure and its inability to
exploit additional resources (beyond those needed for convergence).
Freund and Mason (1999) described a new type of classification mechanism,
the alternating decision tree (ADTree). ADTree provides a method for visualizing
decision stumps in an ordered and logical way to demonstrate correlations.
Freund and Mason presented an iterative algorithm for learning ADTrees, based
on boosting. While it has not been studied as an anytime algorithm, ADTrees
can be viewed as such. The learner starts with a constant prediction and adds
one decision stump at a time. If stopped, it returns the current tree. The anytime
behavior of ADTree, however, is problematic. Additional time resources can only
be used to add more rules and therefore might result in large and over complicated
trees. Moreover, ADtree is not designed to tackle the problem of attribute
interdependencies because it evaluates each split independently.
Murthy, Kasif, and Salzberg (1994) introduced OC1 (Oblique Classifier 1), a
new algorithm for induction of oblique decision trees. Oblique decision trees use
multivariate tests that are not necessarily parallel to an axis. OC1 builds decision
trees that contain linear combinations of one or more attributes at each internal
node; these trees then partition the space of examples with both oblique and axisparallel
hyperplanes. The problem of searching for the best oblique split is much
more difficult than that of searching for the best axis-parallel split because the
number of candidate oblique splits is exponential. Therefore, OC1 takes a greedy
approach and attempts to find locally good splits rather optimal ones. Our LSID3
algorithm can be generalized to support the induction of oblique decision trees by
using OC1 as the sampler, and by considering oblique splits. Furthermore, OC1
can be converted into an anytime algorithm by considering more splits at each
125