anytime algorithms for learning anytime classifiers saher ... - Technion

Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Procedure LSID3-Choose-Attribute(E, A, r)
If r = 0
Return ID3-Choose-Attribute(E, A)
Foreach a ∈ A
Foreach vi ∈ domain(a)
Ei ← {e ∈ E | a(e) = vi}
mini ← ∞
Repeat r times
T ← SID3(Ei, A − {a})
mini ← min (mini, |T |)
mini
Return a for which totala is minimal
totala ← � |domain(a)|
i=1
Figure B.3: Attribute selection in LSID3
accurate. Therefore, LSID3 is expected to improve with the increase in r. Figure
B.3 lists the procedure for attribute selection as applied by LSID3. Because our
goal is to sample small trees and not to always obtain the smallest tree, we use
LSID3(r = 1) as a sampler. Observe that LSID3 is stochastic by nature, and
therefore we do not need to randomize its decisions.
B.3 Empirical Evaluation
We tested Occam’s empirical principle, as stated in Definition 1, on 20 datasets, 18
of which were chosen arbitrarily from the UCI repository (Asuncion & Newman,
2007), and 2 which are artificial datasets that represent hard concepts: XOR-
5 with 5 additional irrelevant attributes, and 20-bit Multiplexer. Each dataset
was partitioned into 10 subsets that were used to create 10 learning problems.
Each problem consisted of one subset serving as a testing set and the union of
the remaining 9 as a training set, as in 10-fold cross validation. We sampled
the version space, TDIDT A(E), for each training set E using the three methods
described in Section B.2.2 and tested the correlation between the size of a tree
(number of leaves) and its accuracy on the associated testing set. The size of the
sample was ten thousand for RTG and SID3, and one thousand for LSID3 (due
to its higher costs). We first present and discuss the results for consistent trees
and then we address the problem of pruned, inconsistent trees.
148