Foundations of Data Science

Boosting Algorithm
Given a sample S of n labeled examples x 1 , . . . , x n , initialize each
example x i to have a weight w i = 1. Let w = (w 1 , . . . , w n ).
For t = 1, 2, . . . , t 0 do
Call the weak learner on the weighted sample (S, w), receiving
hypothesis h t .
End
Multiply the weight of each example that was misclassified by
h t by α = 1 2 +γ
1 . Leave the other weights as they are.
−γ
2
Output the classifier MAJ(h 1 , . . . , h t0 ) which takes the majority vote
of the hypotheses returned by the weak learner. Assume t 0 is odd so
there is no tie.
Figure 6.6: The boosting algorithm
Definition 6.4 (γ-Weak learner on sample) A weak learner is an algorithm that given
examples, their labels, and a nonnegative real weight w i on each example x i , produces a
classifier that correctly labels a subset of examples with total weight at least ( 1 + γ) ∑ n w
2 i .
At the high level, boosting makes use of the intuitive notion that if an example was
misclassified, one needs to pay more attention to it. The boosting procedure is in Figure
6.6.
i=1
Theorem 6.20 Let A be a γ-weak learner for sample S. Then t 0 = O( 1
γ 2 log n) is sufficient
so that the classifier MAJ(h 1 , . . . , h t0 ) produced by the boosting procedure has training
error zero.
Proof: Suppose m is the number of examples the final classifier gets wrong. Each of
these m examples was misclassified at least t 0 /2 times so each has weight at least α t0/2 .
Thus the total weight is at least mα t0/2 . On the other hand, at time t+1, only the weights
of examples misclassified at time t were increased. By the property of weak learning, the
total weight of misclassified examples is at most ( 1 − γ) of the total weight at time t. Let
2
weight(t) be the total weight at time t. Then
(
weight(t + 1) ≤ α ( 1
− γ) + ( ) 1
+ γ) × weight(t)
2 2
= (1 + 2γ) × weight(t).
216