Machine Learning - DISCo

of the difference in their errors
where L(S) denotes the hypothesis output by learning method L when given
the sample S of training data and where the subscript S c V indicates that
the expected value is taken over samples S drawn according to the underlying
instance distribution V. The above expression describes the expected value of the
difference in errors between learning methods LA and L B.
Of course in practice we have only a limited sample Do of data when
comparing learning methods. In such cases, one obvious approach to estimating
the above quantity is to divide Do into a training set So and a disjoint test set To.
The training data can be used to train both LA and LB, and the test data can be
used to compare the accuracy of the two learned hypotheses. In other words, we
measure the quantity
Notice two key differences between this estimator and the quantity in Equation
(5.14). First, we are using errorTo(h) to approximate errorv(h). Second, we
are only measuring the difference in errors for one training set So rather than taking
the expected value of this difference over all samples S that might be drawn
from the distribution 2).
One way to improve on the estimator given by Equation (5.15) is to repeatedly
partition the data Do into disjoint training and test sets and to take the mean
of the test set errors for these different experiments. This leads to the procedure
shown in Table 5.5 for estimating the difference between errors of two learning
methods, based on a fixed sample Do of available data. This procedure first partitions
the data into k disjoint subsets of equal size, where this size is at least 30. It
then trains and tests the learning algorithms k times, using each of the k subsets
in turn as the test set, and using all remaining data as the training set. In this
way, the learning algorithms are tested on k independent test sets, 'and the mean
difference in errors 8 is returned as an estimate of the difference between the two
learning algorithms.
The quantity 8 returned by the procedure of Table 5.5 can be taken as an
estimate of the desired quantity from Equation 5.14. More appropriately, we can
view 8 as an estimate of the quantity
where S represents a random sample of size ID01 drawn uniformly from Do.
The only difference between this expression and our original expression in Equation
(5.14) is that this new expression takes the expected value over subsets of
the available data Do, rather than over subsets drawn from the full instance distribution
2).