anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Procedure SID3-Choose-Attribute(E, A)
Foreach a ∈ A
p (a) ← gain-1(E, a)
If ∃a such that entropy-1(E, a) = 0
a ∗ ← Choose attribute at random from
{a ∈ A | entropy-1(E, a) = 0}
Else
a ∗ ← Choose attribute at random from A;
for each attribute a, the probability
of selecting it is proportional to p (a)
Return a ∗
Figure 3.4: Attribute selection in SID3
Next, we compared the average minimum found for samples of different sizes.
Figure 3.6 shows the results. For the three datasets, the minimal size found by
SID3 is strictly smaller than the value found by RTG. Given the same budget
of time, RTG produced, on average, samples that are twice as large as that of
SID3. However, even when the results are normalized (dashed line), SID3 is still
superior.
Having decided about the sampler, we are ready to describe our proposed
contract algorithm, Lookahead-by-Stochastic-ID3 (LSID3). In LSID3, each candidate
split is evaluated by the estimated size of the subtree under it. To estimate
the size under an attribute a, LSID3 partitions the set of examples according to
the values a can take and repeatedly invokes SID3 to sample the space of trees
consistent with each subset. Summing up the minimal tree size for each subset
gives an estimation of the minimal total tree size under a.
LSID3 is a contract algorithm parameterized by r, the sample size. LSID3
with r = 0 is defined to choose the splitting attribute using the standard ID3
selection method. Figure 3.7 illustrates the choice of splitting attributes as made
by LSID3. In the given example, SID3 is called twice for each subset and the
evaluation of the examined attribute a is the sum of the two minima: min(4, 3)+
min(2, 6) = 5. The method for choosing a splitting attribute is formalized in
Figure 3.8.
To analyze the time complexity of LSID3, let m be the total number of examples
and n be the total number of attributes. For a given node y, we denote by
ny the number of candidate attributes at y, and by my the number of examples
that reach y. ID3, at each node y, calculates gain for ny attributes using my
examples, i.e., the complexity of choosing an attribute is O(ny · my). At level i
of the tree, the total number of examples is bounded by m and the number of
26