anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
in depth k but does not give a direct estimation of the resulting tree size. Thus,
when k < |A|, this heuristic is more informed than entropy1 but can still produce
misleading results. In most cases we do not know in advance what value of k
would be sufficient for correctly learning the target concept. Invoking exhaustive
lookahead, i.e., lookahead to depth k = |A|, will obviously lead to optimal
splits, but its computational costs are prohibitively high. In the following section,
we propose an alternative approach for evaluating attributes that overcomes the
above-mentioned drawbacks of ID3-k.
3.5 A Contract Algorithm for Learning Decision
trees
Motivated by the advantages of smaller decision trees, we introduce a novel algorithm
that, given an attribute a, evaluates it by estimating the size of the minimal
tree under it. The estimation is based on Monte-Carlo sampling of the space of
consistent trees rooted at a. We estimate the minimum by the size of the smallest
tree in the sample. The number of trees in the sample depends on the available
resources, where the quality of the estimation is expected to improve with the
increased sample size.
One way to sample the space of trees is to repeatedly produce random consistent
trees using the RTG procedure. Since the space of consistent decision trees is
large, such a sample might be a poor representative and the resulting estimation
inaccurate. We propose an alternative sampling approach that produces trees of
smaller expected size. Such a sample is likely to lead to a better estimation of
the minimum.
Our approach is based on a new tree generation algorithm that we designed,
called Stochastic ID3 (SID3). Using this algorithm repeatedly allows the space
of “good” trees to be sampled semi-randomly. In SID3, rather than choose an
attribute that maximizes the information gain, we choose the splitting attribute
randomly. The likelihood that an attribute will be chosen is proportional to its
information gain. 3 However, if there are attributes that decrease the entropy to
zero, then one of them is picked randomly. The attribute selection procedure of
SID3 is listed in Figure 3.4.
To show that SID3 is a better sampler than RTG, we repeated our sampling
experiment (Section 3.3) using SID3 as the sample generator. Figure 3.5 compares
the frequency curves of RTG and SID3. Relative to RTG, the graph for
SID3 is shifted to the left, indicating that the SID3 trees are clearly smaller.
3 We make sure that attributes with gain of zero will have a positive probability of being
selected.
25