anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Average Size
Average Accuracy
200
180
160
140
120
100
k=3
r=2
80
0 0.2 0.4 0.6 0.8 1 1.2
92
90
88
86
84
82
80
78
r=2
k=3
Time [seconds]
r=10
LSID3
ID3k
ID3
C4.5
0 0.2 0.4 0.6 0.8 1 1.2
Time [seconds]
r=10
LSID3
ID3k
ID3
C4.5
Figure 3.25: Anytime behavior of ID3-k and LSID3 on the Tic-tac-toe dataset
ID3-k. LSID3 clearly outperforms all the other algorithms and exhibits good
anytime behavior. Generalization accuracy and tree size both improve with time.
ID3-k behaves poorly in this case. For example, when 200 seconds are allocated,
we can run LSID3 with r = 2 and achieve accuracy of about 90%. With the same
allocation, ID3-k can be run with k = 2 and achieve accuracy of about 52%.
The next improvement of ID3-k (with k = 3) requires 10,000 seconds. But even
with such a large allocation (not shown in the graph since it is off the scale), the
resulting accuracy is only about 66%.
In Section 3.5.1 we described the LSID3-MC algorithm which, instead of uniformly
distributing evaluation resources over all possible splitting points, performs
biased sampling towards points with high information gain. Figure 3.27 compares
the anytime behavior of LSID3-MC to that of LSID3. The graph of LSID3 shows,
as before, the performance for successive values of r. The graph of LSID3-MC
shows the performance for p = 10%, 20%, . . ., 150%. A few significant conclusions
can be drawn from these results:
57