anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Chapter 2
Resource-bounded Learning and
Classification
In real-life applications, inductive concept learning involves several types of cost
(Turney, 2000). In the learning phase costs are associated with the acquisition
of feature-values (Melville, Saar-Tsechansky, Provost, & Mooney, 2004; Lizotte
et al., 2003; Sheng & Ling, 2007a) and with tagging examples (Lindenbaum
et al., 2004).
Once the data has been collected, the next stage is model learning. This
stage usually requires resources such as memory and computation time. Algorithms
whose results improve with the increase in the allocated resources are
called anytime algorithms. Anytime algorithms are attractive when optimal solutions
require unaffordable resources and quick greedy solutions do not suffice.
Similar situations, where the solution can be either of exponential complexity
and therefore prohibitively expensive or non-optimal and fast, are common in AI:
anytime techniques have been applied to several AI domains, such as heuristic
search (Hansen, Zilberstein, & Danilchenko, 1997), constraint satisfaction problems
(Fargier, Lang, & Schiex, 1996), and SAT solvers (Wallace & Freuder, 1995).
There are two main classes of anytime algorithms, namely contract and interruptible
(Russell & Zilberstein, 1996). A contract algorithm is one that gets
its resource allocation as a parameter. If interrupted at any point before the
allocation is completed, it might not yield any useful results. An interruptible algorithm
is one whose resource allocation is not given in advance and thus must be
prepared to be interrupted at any moment. While the assumption of preallocated
resources holds for many induction tasks, in many other real-life applications it
is not possible to allocate the resources in advance. Therefore, in this work we
are interested in both types of anytime algorithms.
During the classification phase, costs are associated with the measurements
required by the model and with the prediction errors. Following previous works
13