anytime algorithms for learning anytime classifiers saher ... - Technion

More documents

Recommendations

Info

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008 original costs and the contribution of each attribute in reducing misclassification costs. Then it builds a tree using the EG2 algorithm (Nunez, 1991) but with the evolved costs instead of the original ones. 7 ICET has been reimplemented following the detailed description given by Turney (1995). To verify the results of the reimplementation, we compared them with those reported in the literature. We followed the same experimental setup and used the same 5 datasets. The results are indeed similar: the basic version of ICET achieved an average cost of 50.8 in our reimplementation vs. 50 reported originally. One possible reason for the slight difference may be that the initial population of the genetic algorithm is randomized, as are the genetic operators and the process of partitioning the data into training, validating, and testing sets. In his paper, Turney introduced a seeded version of ICET, which includes the true costs in the initial population, and reported it to perform better than the unseeded version. Therefore, we use the seeded version for our comparison. The other parameters of ICET are the default ones. Normalized Cost As Turney (1995) points out, using the average cost of classification for each dataset is problematic because: (1) the cost differences of the algorithms become relatively small as the misclassification cost increases, (2) it is difficult to combine the results for multiple datasets in a fair manner (e.g., average), and (3) it is difficult to combine the average of the different misclassification costs. To overcome these problems, Turney suggests normalizing the average cost of classification by dividing it by the standard cost. Let TC be the cost if we take all tests. Let fi be the frequency of class i in the data. The error if the response is always class i is therefore (1 − fi). The standard cost is defined as TC + mini (1 − fi) · maxi,j (Mi,j) . The standard cost is an approximation for the maximal cost in a given problem. It consists of two components: the maximal test cost and the misclassification cost if the classifier achieves only the baseline accuracy (e.g., a majority-based classifier when error costs are uniform). Because some classifiers may perform even worse than the baseline accuracy, the standard cost is not strictly an upper bound on real cost. In most of our experiments, however, it has not been exceeded. 7 Detailed description of ICET can be found in Chapter 6. 84
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008 Table 4.2: Average cost of classification as a percentage of the standard cost of classification for different mc values. The numbers represent the average over all 105 datasets and the associated confidence intervals (α = 5%). mc C4.5 LSID3 IDX CSID3 EG2 DTMC ICET ACT 100 50.6 ±4 45.3 ±4 34.4 ±4 41.7 ±4 35.1 ±4 14.6 ±2 24.3 ±3.1 15.2 ±2 500 49.9 ±4 43.0 ±4 42.4 ±4 45.2 ±4 42.5 ±4 37.7 ±3 36.3 ±3.1 34.5 ±3 1000 50.4 ±5 42.4 ±5 47.5 ±4 47.8 ±4 47.3 ±4 47.1 ±4 40.6 ±3.9 39.1 ±4 5000 53.3 ±6 43.6 ±6 58.1 ±6 54.3 ±6 57.3 ±6 57.6 ±5 45.7 ±5.6 41.5 ±6 10000 54.5 ±6 44.5 ±7 60.8 ±6 56.2 ±6 59.9 ±6 59.5 ±6 47.1 ±6.0 41.4 ±6 Statistical Significance For each problem, a single 10-fold cross-validation experiment was conducted. The same partition to train-test sets was used for all compared algorithms. To determine statistical significance of the performance differences between ACT, ICET, and DTMC we used two tests: • A t-test with α = 5% confidence. For each method we counted how many times it was a significant winner. • Wilcoxon test (Demsar, 2006), which compares classifiers over multiple datasets and states whether one method is significantly better than the other (α = 5%). 4.6.3 Fixed-time Comparison For each of the 105 × 5 problem instances, we ran the different algorithms, including ACT with r = 5. We chose r = 5 so the average runtime of ACT would be shorter than ICET over all problems. The other methods have much shorter runtime due to their greedy nature. Table 4.2 summarizes the results and Tables 4.4-4.8 list the detailed results for the different datasets. 8 Each table corresponds to a different mc value. Each pair of numbers represents the average normalized cost and its associated confidence interval (α = 5%). Figure 4.7 illustrates the average results and plots the normalized costs for the different algorithms and misclassification costs. Statistical significance test results for ACT, ICET, and DTMC are given in Table 4.3. The algorithms are compared using both the t-test and the Wilcoxon 8 For the 25 datasets with automatic cost assignment, the average over the 4 seeds is listed. The full results are available at http://www.cs.technion.ac.il/∼esaher/publications/cost. 85
Page 1 and 2:
Technion - Computer Science Departm
Page 3 and 4:
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40:
Page 41 and 42:
Page 43 and 44:
Page 45 and 46:
Page 47 and 48:
Page 49 and 50: Technion - Computer Science Departm
Page 99: Technion - Computer Science Departm
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
Page 187 and 188:
Page 189 and 190:
Page 191 and 192:
show all

anytime algorithms for learning anytime classifiers saher ... - Technion

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?