Foundations of Data Science

Theorem 6.11 (Online to Batch via Random Stopping) If an online algorithm A
with mistake-bound M is run on a sample S of size M/ɛ and stopped at a random time
between 1 and |S|, the expected error of the hypothesis h produced satisfies E[err D (h)] ≤ ɛ.
Conversion procedure 2: Controlled Testing. A second natural approach to using
an online learning algorithm A in the distributional setting is to just run a series of
controlled tests. Specifically, suppose that the initial hypothesis produced by algorithm
A is h 1 . Define δ i = δ/(i + 2) 2 so we have ∑ ∞
i=0 δ i = ( π2 − 1)δ ≤ δ. We draw a set of
6
n 1 = 1 log( 1 ɛ δ 1
) random examples and test to see whether h 1 gets all of them correct. Note
that if err D (h 1 ) ≥ ɛ then the chance h 1 would get them all correct is at most (1−ɛ) n 1
≤ δ 1 .
So, if h 1 indeed gets them all correct, we output h 1 as our hypothesis and halt. If not,
we choose some example x 1 in the sample on which h 1 made a mistake and give it to
algorithm A. Algorithm A then produces some new hypothesis h 2 and we again repeat,
testing h 2 on a fresh set of n 2 = 1 log( 1 ɛ δ 2
) random examples, and so on.
In general, given h t we draw a fresh set of n t = 1 ɛ log( 1 δ t
) random examples and test
to see whether h t gets all of them correct. If so, we output h t and halt; if not, we choose
some x t on which h t (x t ) was incorrect and give it to algorithm A. By choice of n t , if h t
had error rate ɛ or larger, the chance we would mistakenly output it is at most δ t . By
choice of the values δ t , the chance we ever halt with a hypothesis of error ɛ or larger is at
most δ 1 + δ 2 + . . . ≤ δ. Thus, we have the following theorem.
Theorem 6.12 (Online to Batch via Controlled Testing) Let A be an online learning
algorithm with mistake-bound M. Then this procedure will halt after O( M ɛ log( M δ ))
examples and with probability at least 1 − δ will produce a hypothesis of error at most ɛ.
Note that in this conversion we cannot re-use our samples: since the hypothesis h t depends
on the previous data, we need to draw a fresh set of n t examples to use for testing it.
6.8 Support-Vector Machines
In a batch setting, rather than running the Perceptron algorithm and adapting it via
one of the methods above, another natural idea would be just to solve for the vector w
that minimizes the right-hand-side in Theorem 6.9 on the given dataset S. This turns
out to have good guarantees as well, though they are beyond the scope of this book. In
fact, this is the Support Vector Machine (SVM) algorithm. Specifically, SVMs solve the
following convex optimization problem over a sample S = {x 1 , x 2 , . . . x n } where c is a
constant that is determined empirically.
minimize
c|w| 2 + ∑ i
s i
subject to
w · x i ≥ 1 − s i for all positive examples x i
w · x i ≤ −1 + s i for all negative examples x i
s i ≥ 0 for all i.
205