anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
population and to produce more generations. The accuracy on the testing set,
even after thousands of generations, remained very low. Similar results were
obtained for the Multiplexers-20 dataset.
The above set of experiments was repeated on the much more difficult XOR-10
dataset. The advantage of IIDT over the other methods was even more evident.
While IIDT was able to reach accuracy of 100%, bagging-ID3, skewing, and
GATree performed as poorly as a random guesser, with accuracy of only 50%.
The next experiment was with the Tic-tac-toe dataset. In this case, as shown
in Figure 3.35, both ensemble-based methods have a significant advantage over
the single tree inducers. We speculate that this is because ensemble methods
were able to overcome the quick-fragmentation problem associated with multiway
splits by combining several classifiers. We are still looking for ways to verify
this hypothesis. Bagging-ID3 outperforms the other methods until the fifth second,
where bagging-LSID3 overtakes it slightly. In contrast to the XOR-5 domain,
building larger committees is worthwhile in this case, even at the expense of less
accurate base classifiers. However, if the time allocation permits, large ensembles
of LSID3 trees are shown to be the most accurate. We believe that the general
question of tradeoff between the resources allocated for each tree and the number
of trees forming the ensemble should be addressed by further research with extensive
experiments on various datasets. The performance of generalized skewing
and IIDT was similar in this case, with a slight advantage for skewing in terms
of accuracy and an advantage for IIDT in terms of tree size. GATree was run on
the dataset for 150 generations (30 seconds). The average accuracy was 76.42%,
much lower than that of the other methods.
69