anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Table 3.6: The average differences between the accuracy of the different algorithms
and their t-test significance, with α = 0.05 ( √ indicates a significant advantage and
× a significant disadvantage). The t-test is not applicable for the Monk datasets
because only 1 train-test partition was used.
LSID3 vs. ID3 LSID3 vs. C4.5 LSID3-P vs. C4.5
Dataset Diff Sig? Diff Sig? Diff Sig?
Autos Make
Autos Sym.
Balance
1.2 ±9.9
-0.6 ±11.4
1.5 ±4.5
∼
√
∼
0.8 ±10.3
3.9 ±12.3
6.5 ±5.6
√
∼
√
-0.2 ±10.6
3.7 ±12.2
4.1 ±5.5
√
∼
√
Br. Cancer
Connect-4
Corral
Glass
Iris
-0.2 ±9.1
3.0 ±0.6
13.8 ±25.4
4.3 ±11.6
1.1 ±4.0
√
∼
√
√
√
-7.8 ±9.1
-2.7 ±0.5
18.2 ±25.1
3.0 ±12.7
-0.8 ±7.8
×
√
×
√
∼
-3.6 ±6.0
-0.9 ±0.4
14.2 ±24.1
2.0 ±11.5
-0.9 ±7.9
×
√
×
∼
∼
Monks-1 17.1 ±0.0 - 24.3 ±0.0 - 18.8 ±0.0 -
Monks-2 -2.2 ±0.5 - 2.0 ±0.5 - -1.5 ±1.1 -
Monks-3 -3.0 ±1.5 - -5.7 ±1.5 - 0.8 ±1.3 -
Mushroom 0.0 ±0.0 ∼ 0.0 ±0.0 ∼ 0.0 ±0.0 ∼
Solar Flare
Tic-tac-toe
-0.2 ±3.0
1.5 ±4.5
√
∼ -2.4 ±8.0
1.2 ±4.8
√
× 0.0 ±7.6
1.4 ±4.6
√
∼
Voting 0.4 ±4.6 ∼ -0.9 ±4.4 ∼ 0.1 ±1.8 ∼
Wine -0.1 ±7.4 ∼ -0.4 ±8.7 ∼ -0.6 ±10.0 ∼
Zoo
Numeric XOR-3D
Numeric XOR-4D
Multiplexer-20
Multiplex-XOR
XOR-5
XOR-5 Noise
XOR-10
-1.0 ±8.0
38.4 ±12.0
43.4 ±7.8
34.1 ±11.8
18.3 ±16.2
44.5 ±12.2
20.3 ±17.2
27.5 ±14.7
√
∼
√
√
√
√
√
√
2.1 ±10.7
53.1 ±7.1
41.1 ±6.2
33.4 ±11.0
21.1 ±15.1
45.9 ±12.8
22.7 ±15.9
27.7 ±14.8
√
∼
√
√
√
√
√
√
1.3 ±10.1
50.4 ±7.6
39.8 ±5.9
32.3 ±12.1
24.7 ±14.7
45.9 ±12.8
26.5 ±18.2
29.6 ±16.7
√
∼
√
√
√
√
√
√
C4.5’s pruning are different. While LSID3 allots more time for better learning of
hard concepts, pruning attempts to simplify the induced trees to avoid overfitting
the data. The combination of LSID3 and pruning is shown to be worthwhile: it
enjoys the benefits associated with lookahead without the need to compromise
when the training set is noisy.
We also examined the effect of applying error-based pruning not only to the
final tree, but to the lookahead trees as well. The experiments conducted on
several noisy datasets showed that the results of this extra pruning phase were
very similar to the results without pruning. Although pruning results in samples
that better represent the final trees, it affects all samples similarly and hence does
not lead to different split decisions.
52