Foundations of Data Science

work on a variety of measures. One measure called Rademacher complexity measures
the extent to which a given concept class H can fit random noise. Given a set of n
examples S = {x 1 , . . . ,
∑
x n }, the empirical Rademacher complexity of H is defined as
1
R S (H) = E σ1 ,...,σ n
max n
h∈H
n i=1 σ ih(x i ), where σ i ∈ {−1, 1} are independent random labels
with Prob[σ i = 1] = 1 . E.g., if you assign random ±1 labels to the points in S and the
2
best classifier in H on average gets error 0.45 then R S (H) = 0.55 − 0.45 = 0.1. One can
prove that with probability greater than or equal to 1 − δ, every h ∈ H satisfies true error
√
less than or equal to training error plus R S (H) + 3
this, see, e.g., [?].
6.10 Strong and Weak Learning - Boosting
ln(2/δ)
. For more on results such as
2n
We now describe boosting, which is important both as a theoretical result and as a
practical and easy-to-use learning method.
A strong learner for a problem is an algorithm that with high probability is able to
achieve any desired error rate ɛ using a number of samples that may depend polynomially
on 1/ɛ. A weak learner for a problem is an algorithm that does just a little bit better than
random guessing. It is only required to get with high probability an error rate less than
or equal to 1 − γ for some 0 < γ ≤ 1 . We show here that a weak-learner for a problem
2 2
that achieves the weak-learning guarantee for any distribution of data can be boosted to a
strong learner, using the technique of boosting. At the high level, the idea will be to take
our training sample S, and then to run the weak-learner on different data distributions
produced by weighting the points in the training sample in different ways. Running the
weak learner on these different weightings of the training sample will produce a series of
hypotheses h 1 , h 2 , . . ., and the idea of our reweighting procedure will be to focus attention
on the parts of the sample that previous hypotheses have performed poorly on. At the
end we will combine the hypotheses together by a majority vote.
Assume the weak learning algorithm A outputs hypotheses from some class H. Our
boosting algorithm will produce hypotheses that will be majority votes over t 0 hypotheses
from H, for t 0 defined below. This means that we can apply Corollary 6.19 to bound the
VC-dimension of the class of hypotheses our boosting algorithm can produce in terms of
the VC-dimension of H. In particular, the class of rules that can be produced by the
booster running for t 0 rounds has VC-dimension O(t 0 VCdim(H) log(t 0 VCdim(H))). This
in turn gives a bound on the number of samples needed, via Corollary 6.16, to ensure that
high accuracy on the sample will translate to high accuracy on new data.
To make the discussion simpler, we will assume that the weak learning algorithm A,
when presented with a weighting of the points in our training sample, always (rather than
with high probability) produces a hypothesis that performs slightly better than random
guessing with respect to the distribution induced by weighting. Specificially:
215