anytime algorithms for learning anytime classifiers saher ... - Technion

More documents

Recommendations

Info

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008 Table 4.3: DTMC vs. ACT and ICET vs. ACT using statistical tests. For each mc, the first column lists the number of t-test significant wins while the second column gives the winner, if any, as implied by a Wilcoxon test over all datasets with α = 5%. t − test WINS Wilcoxon WINNER mc DTMC vs. ACT ICET vs. ACT DTMC vs. ACT ICET vs. ACT 100 14 3 4 54 DTMC ACT 500 9 29 5 23 ACT ACT 1000 7 45 12 24 ACT ACT 5000 7 50 15 21 ACT ACT 10000 6 56 7 24 ACT - Average % Standard Cost 60 50 40 30 C4.5 LSID3 EG2 20 DTMC ICET 10 ACT 100 1000 10000 Misclassification Cost Figure 4.7: Average normalized cost as a function of misclassification cost test. The table lists the number of t-test wins for each algorithm out of the 105 datasets, as well as the winner, if any, when the Wilcoxon test was applied. When misclassification cost is relatively small (mc = 100), ACT clearly outperforms ICET, with 54 significant wins as opposed to ICET’s 4 significant wins. No significant difference was found in the remaining runs. In this setup ACT was able to produce very small trees, sometimes consisting of one node; the accuracy of the learned model was ignored in this setup. ICET, on the contrary, produced, for some of the datasets, larger and more costly trees. DTMC achieved the best results, and outperformed ACT 14 times. The Wilcoxon test also indicates that DTMC is better than ACT and that ACT is better than ICET. Further investigation showed that for a few datasets ACT produced unnecessarily larger trees. 86
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008 Table 4.4: Average cost of classification as a percentage of the standard cost of classification for mc = 100. The first 25 rows list for each dataset the average cost over the different 4 cost-assignments, while the last 5 rows give the results for the datasets with costs from (Turney, 1995). Dataset C4.5 LSID3 IDX CSID3 EG2 DTMC ICET ACT Breast-cancer 28.9 22.7 14.3 25.9 14.6 7.1 9.6 7.1 Bupa 88.7 74.3 55.5 75.6 56.6 15.7 20.9 15.7 Car 62.2 64.4 57.6 60.5 57.6 11.8 43.0 11.8 Flare 3.4 3.4 6.1 4.1 6.1 3.4 3.8 3.4 Glass 54.3 47.1 17.4 25.5 16.8 15.8 16.0 16.1 Heart 38.5 38.1 22.7 32.3 24.0 8.7 14.3 8.6 Hepatitis 26.8 20.9 9.2 17.5 10.2 3.1 5.4 3.1 Iris 50.8 46.3 40.4 40.5 39.6 19.1 29.1 22.5 KRK 73.6 67.9 59.7 63.1 60.4 23.4 50.4 23.4 Multi-ANDOR 46.2 25.7 25.8 29.6 26.5 11.6 13.1 12.5 Monks1 52.1 40.8 50.4 52.1 52.0 26.2 45.2 27.6 Monks2 21.6 49.6 34.0 28.0 31.2 11.9 11.9 11.9 Monks3 58.1 55.6 53.8 54.0 53.8 17.7 51.3 17.7 Multiplexer 44.1 24.2 21.3 31.5 22.1 10.9 10.9 10.7 MultiXOR 56.8 39.3 29.5 42.7 30.7 13.2 13.1 13.3 Nursery 51.8 55.2 51.6 50.6 50.4 23.8 49.2 23.8 Pima 70.5 72.1 35.7 63.2 40.7 12.4 16.2 12.5 Tae 64.3 52.0 47.4 54.2 51.0 28.1 34.4 27.1 Tic-tac-toe 60.4 56.1 30.7 51.4 32.1 13.5 25.3 13.5 Titanic 85.3 53.4 67.6 72.2 68.2 18.4 48.2 18.4 Thyroid 26.6 25.6 27.6 26.3 26.3 2.1 20.8 2.1 Voting 16.8 17.8 13.3 16.5 13.9 5.6 13.4 7.0 Wine 31.7 31.4 11.0 18.2 12.2 7.8 10.1 8.6 XOR3d 82.2 66.1 29.4 57.8 30.9 22.0 22.5 24.1 XOR5 56.0 64.0 38.9 45.4 39.9 12.4 12.4 12.4 Bupa 89.8 85.2 88.7 86.7 88.7 65.1 66.5 76.2 Heart 58.6 57.0 8.5 11.7 8.8 8.2 8.0 8.4 Hepatitis 53.8 48.6 31.3 35.3 34.0 26.8 36.4 34.8 Pima 65.8 71.4 52.0 54.4 52.0 44.7 45.7 46.6 Thyroid 33.6 33.5 32.5 32.9 32.3 9.7 28.3 9.7 We believe that a better tuning of cf would improve ACT in this scenario by making the pruning more aggressive. At the other extreme, when misclassification costs dominate (mc = 10000), the performance of DTMC is worse than ACT and ICET. The t-test indicates that ACT was significantly better than ICET 24 times and significantly worse only 7 times. According to the Wilcoxon test with α = 5%, the difference between ACT and ICET is not significant. Taking α > 5.05%, however, would turn the 87
Page 1 and 2:
Technion - Computer Science Departm
Page 3 and 4:
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40:
Page 41 and 42:
Page 43 and 44:
Page 45 and 46:
Page 47 and 48:
Page 49 and 50:
Page 51 and 52: Technion - Computer Science Departm
Page 101: Technion - Computer Science Departm
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
Page 187 and 188:
Page 189 and 190:
Page 191 and 192:
show all

anytime algorithms for learning anytime classifiers saher ... - Technion

Create successful ePaper yourself

Delete template?

Save as template?