Foundations of Data Science

imply that if it does and if S is sufficiently large, we can be confident that its true error
will be low as well. Suppose that stochastic gradient descent is run on a machine where
each weight is a 64-bit floating point number. This means that its hypotheses can each
be described using 64n bits. If S has size at least 1 [64n ln(2) + ln(1/δ)], by Theorem 6.5
ɛ
it is unlikely any such hypothesis of true error greater than ɛ will be consistent with the
sample, and so if it finds a hypothesis consistent with S, we can be confident its true error
is at most ɛ. Or, by Theorem 6.3, if |S| ≥ 1
2ɛ 2 (
64n ln(2) + ln(2/δ)
)
then almost surely the
final hypothesis h produced by stochastic gradient descent satisfies true error leas than
or equal to training error plus ɛ.
6.12 Combining (Sleeping) Expert Advice
Imagine you have access to a large collection of rules-of-thumb that specify what to
predict in different situations. For example, in classifying news articles, you might have
one that says “if the article has the word ‘football’, then classify it as sports” and another
that says “if the article contains a dollar figure, then classify it as business”. In predicting
the stock market, these could be different economic indicators. These predictors might
at times contradict each other, e.g., a news article that has both the word “football” and
a dollar figure, or a day in which two economic indicators are pointing in different directions.
It also may be that no predictor is perfectly accurate with some much better than
others. We present here an algorithm for combining a large number of such predictors
with the guarantee that if any of them are good, the algorithm will perform nearly as well
as each good predictor on the examples on which that predictor fires.
Formally, define a “sleeping expert” to be a predictor h that on any given example x
either makes a prediction on its label or chooses to stay silent (asleep). We will think of
them as black boxes. Now, suppose we have access to n such sleeping experts h 1 , . . . , h n ,
and let S i denote the subset of examples on which h i makes a prediction (e.g., this could
be articles with the word “football” in them). We consider the online learning model,
and let mistakes(A, S) denote the number of mistakes of an algorithm A on a sequence
of examples S. Then the guarantee of our algorithm A will be that for all i
E ( mistakes(A, S i ) ) ≤ (1 + ɛ) · mistakes(h i , S i ) + O ( log n
ɛ
where ɛ is a parameter of the algorithm and the expectation is over internal randomness
in the randomized algorithm A.
As a special case, if h 1 , . . . , h n are concepts from a concept class H, and so they all
make predictions on every example, then A performs nearly as well as the best concept
in H. This can be viewed as a noise-tolerant version of the Halving Algorithm of Section
6.5.2 for the case that no concept in H is perfect. The case of predictors that make
predictions on every example is called the problem of combining expert advice, and the
more general case of predictors that sometimes fire and sometimes are silent is called the
)
220