Foundations of Data Science

then with probability ≥ 1 − δ, every h ∈ H will have |err S (h) − err D (h)| ≤ ɛ.
Proof: This proof is identical to the proof of Theorem 6.13 except B ∗ is now the event
that there exists a set h ∈ H[S ′′ ] such that the error of h on S differs from the error of h on
S ′ by more than ɛ/2. We again consider the experiment where we randomly put the points
in S ′′ into pairs (a i , b i ) and then flip a fair coin for each index i, if heads placing a i into S
and b i into S ′ , else placing a i into S ′ and b i into S. Consider the difference between the
number of mistakes h makes on S and the number of mistakes h makes on S ′ and observe
how this difference changes as we flip coins for i = 1, 2, . . . , n. Initially, the difference
is zero. If h makes a mistake on both or neither of (a i , b i ) then the difference does not
change. Else, if h makes a mistake on exactly one of a i or b i , then with probability 1/2
the difference increases by one and with probability 1/2 the difference decreases by one.
If there are r ≤ n such pairs, then if we take a random walk of r ≤ n steps, what is the
probability that we end up more than ɛn/2 steps away from the origin? This is equivalent
to asking: if we flip r ≤ n fair coins, what is the probability the number of heads differs
from its expectation by more than ɛn/4. By Hoeffding bounds, this is at most 2e −ɛ2 n/8 .
This quantity is at most δ/(2H[2n]) as desired for n as given in the theorem statement.
Finally, we prove Sauer’s lemma, relating the growth function to the VC-dimension.
Theorem 6.15 (Sauer’s lemma) If VCdim(H) = d then H[n] ≤ ∑ d
( n
)
i=0 i ≤ (
en
d )d .
Proof: Let d = VCdim(H). Our goal is to prove for any set S of n points that
|H[S]| ≤ ( (
n
≤d)
, where we are defining
n
∑
≤d)
=
d
( n
i=0 i)
; this is the number of distinct
ways of choosing d or fewer elements out of n. We will do so by induction on n. As a
base case, our theorem is trivially true if n ≤ d.
As a first step in the proof, notice that:
( ) ( )
n n − 1
=
≤ d ≤ d
+
( ) n − 1
≤ d − 1
(6.2)
because we can partition the ways of choosing d or fewer items into those that do not
include the first item (leaving ≤ d to be chosen from the remainder) and those that do
include the first item (leaving ≤ d − 1 to be chosen from the remainder).
Now, consider any set S of n points and pick some arbitrary point x ∈ S. By induction,
we may assume that |H[S \ {x}]| ≤ ( )
n−1
≤d . So, by equation (6.2) all we need to show
is that |H[S]| − |H[S \ {x}]| ≤ ( n−1
≤d−1)
. Thus, our problem has reduced to analyzing how
many more partitions there are of S than there are of S \ {x} using sets in H.
If H[S] is larger than H[S \ {x}], it is because of pairs of sets in H[S] that differ only
on point x and therefore collapse to the same set when x is removed. For set h ∈ H[S]
213