Foundations of Data Science

A
B
D
C
(a)
(b)
Figure 6.5: (a) shows a set of four points that can be shattered by rectangles along with
some of the rectangles that shatter the set. Not every set of four points can be shattered
as seen in (b). Any rectangle containing points A, B, and C must contain D. No set of five
points can be shattered by rectangles with axis-parallel edges. No set of three collinear
points can be shattered, since any rectangle that contains the two end points must also
contain the middle point. More generally, since rectangles are convex, a set with one point
inside the convex hull of the others cannot be shattered.
Theorem 6.13 (Growth function sample bound) For any class H and distribution
D, if a training sample S is drawn from D of size
n ≥ 2 ɛ [log 2(2H[2n]) + log 2 (1/δ)]
then with probability ≥ 1−δ, every h ∈ H with err D (h) ≥ ɛ has err S (h) > 0 (equivalently,
every h ∈ H with err S (h) = 0 has err D (h) < ɛ).
Theorem 6.14 (Growth function uniform convergence) For any class H and distribution
D, if a training sample S is drawn from D of size
n ≥ 8 [ln(2H[2n]) + ln(1/δ)]
ɛ2 then with probability ≥ 1 − δ, every h ∈ H will have |err S (h) − err D (h)| ≤ ɛ.
Theorem 6.15 (Sauer’s lemma) If VCdim(H) = d then H[n] ≤ ∑ d
( n
)
i=0 i ≤ (
en
d )d .
Notice that Sauer’s lemma was fairly tight in the case of axis-parallel rectangles,
though in some cases it can be a bit loose. E.g., we will see that for linear separators
in the plane, their VC-dimension is 3 but H[n] = O(n 2 ). An interesting feature about
Sauer’s lemma is that it implies the growth function switches from taking the form 2 n to
taking the form n VCdim(H) when n reaches the VC-dimension of the class H.
Putting Theorems 6.13 and 6.15 together, with a little algebra we get the following
corollary (a similar corollary results by combining Theorems 6.14 and 6.15):
208