anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
The SVM algorithm usually depends on several parameters – kernel parameters,
for example. Several works, such as (Chapelle, Vapnik, Bousquet, & Mukherjee,
2002), proposed iterative methods for automatic tuning of SVM parameters.
These iterative methods can exploit additional time resources for better tuning.
A well-studied alternative to inductive learning is the theory refinement
paradigm. In a theory refinement system, we first acquire a domain theory,
for instance by querying experts, and then revise the obtained set of rules in an
attempt to make it consistent with the training data. Opitz (1995) introduced
an anytime approach for theory refinement. This approach starts by generating
a neural network from a set of rules that describe what is currently known about
the domain. The network then uses the training data and the additional time
resources to try to improve the resulting hypothesis.
6.3 Cost-sensitive Classification
Cost-sensitive trees have been the subject of many research efforts. Several
works proposed learning algorithms that consider different misclassification costs
(Breiman et al., 1984; Pazzani, Merz, Murphy, Ali, Hume, & Brunk, 1994;
Provost & Buchanan, 1995; Bradford, Kunz, Kohavi, Brunk, & Brodley, 1998;
Domingos, 1999b; Drummond & Holte, 2000; Elkan, 2001; Zadrozny, Langford,
& Abe, 2003; Lachiche & Flach, 2003; Abe, Zadrozny, & Langford, 2004; Vadera,
2005; Margineantu, 2005; Zhu, Wu, Khoshgoftaar, & Yong, 2007; Sheng & Ling,
2007b). These methods, however, do not consider test costs and hence are appropriate
mainly for domains where test costs are not a constraint. Other authors
designed tree learners that take into account test costs, such as IDX (Norton,
1989), CSID3 (Tan & Schlimmer, 1989), and EG2 (Nunez, 1991). These methods,
however, do not consider misclassification costs.
Decision Trees with Minimal Cost (DTMC), a greedy method that attempts
to minimize both types of costs simultaneously, has been recently introduced
(Ling et al., 2004; Sheng et al., 2006). A tree is built top-down, and a greedy
split criterion that takes into account both testing and misclassification costs is
used. The basic idea is to estimate the immediate reduction in total cost after
each split, and to prefer the split with the maximal reduction. If no split reduces
the cost on the training data, the induction process is stopped.
Although efficient, the DTMC approach can be trapped into a local minimum
and produce trees that are not globally optimal. For example, consider the concept
and costs described in Figure 6.2 (left). There are 10 attributes, of which
only a9 and a10 are relevant. The cost of a9 and a10, however, is significantly
higher than the others. Such high costs may hide the usefulness of a9 and a10,
and mislead the learner into repeatedly splitting on a1−8, which would result in
128