anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
well and no lookahead is needed. However, for more difficult concepts such as
XOR, the greedy approach is likely to fail. A third problem is their limitedness
to a specific objective: they cannot be adapted to different learning setups and
other objectives, such as minimizing testing and misclassification costs.
We therefore propose an alternative approach for looking ahead. For each
candidate split we sample the spaces of subtrees under it and estimate the utility
of the sampled trees. Because we evaluate entire trees, different utility functions
can be used, depending on the actual cost scheme. The split with the best tree
in its sample is then selected to split on.
In the cost-insensitive setup, our goal is to induce small and accurate trees.
Following Occam’s razor, we bias the sample towards small consistent trees and
evaluate each sample tree by its size. To avoid overfitting the training examples,
we apply a post-pruning phase, similarly to C4.5.
When our objective is to minimize the total cost, we bias the sample towards
low cost trees, and evaluate the sampled trees by their expected total cost. The
total cost of a tree is estimated using the average costs of classifying the training
examples using the tree, and the expected error of the tree. In cost-insensitive
environments, the main goal of pruning is to simplify the tree in order to avoid
overfitting the training data. A subtree is pruned if the resulting tree is expected
to yield a lower error. When test costs are taken into account, pruning has another
important role: reducing test costs in a tree. Keeping a subtree is worthwhile
only if its expected reduction in misclassification costs is larger than the cost of
the tests in that subtree. Therefore, we designed a novel pruning approach based
on the expected total cost of a tree.
For the scenarios that constrain the testing costs, we developed a novel topdown
approach to exploit the available testing resources. When the bounds are
known to the learner, a tree that fits the budget is built. In other cases, a
repertoire of trees is formed. If the quota is known before classification, a single
tree that best fits the budget is picked. Otherwise, the trees are traversed until
resources are exhausted.
Our anytime approach can benefit from extra learning time by creating larger
samples. The larger the samples are, the more accurate the attribute evaluation
is. There are two main classes of anytime algorithms, namely contract and interruptible
(Russell & Zilberstein, 1996). A contract algorithm is one that gets
its resource allocation as a parameter. An interruptible algorithm is one whose
resource allocation is not given in advance and thus must be prepared to be interrupted
at any moment. While the assumption of preallocated resources holds
for many induction tasks, in many other real-life applications it is not possible to
allocate the resources a priori. Therefore, in our work, we are interested both in
contract and interruptible decision tree learners. In the contract setup, the sample
size is predetermined according to the available resources. In the interruptible
9