Foundations of Data Science

A sphere in d-dimensions is a set of points of the form {x| |x − x 0 | ≤ r}. The VCdimension
of spheres is d + 1. It is the same as that of half spaces. First, we prove that
no set of d + 2 points can be shattered by spheres. Suppose some set S with d + 2 points
can be shattered. Then for any partition A 1 and A 2 of S, there are spheres B 1 and B 2
such that B 1 ∩ S = A 1 and B 2 ∩ S = A 2 . Now B 1 and B 2 may intersect, but there is no
point of S in their intersection. It is easy to see that there is a hyperplane perpendicular
to the line joining the centers of the two spheres with all of A 1 on one side and all of A 2
on the other and this implies that half spaces shatter S, a contradiction. Therefore no
d + 2 points can be shattered by hyperspheres.
It is also not difficult to see that the set of d+1 points consisting of the unit-coordinate
vectors and the origin can be shattered by spheres. Suppose A is a subset of the d + 1
points. Let a be the number of unit vectors in A. The center a 0 of our sphere will be
the sum of the vectors in A. For every unit vector in A, its distance to this center will
be √ a − 1 and for every unit vector outside A, its distance to this center will be √ a + 1.
The distance of the origin to the center is √ a. Thus, we can choose the radius so that
precisely the points in A are in the hypersphere.
Finite sets
The system of finite sets of real numbers can shatter any finite set of real numbers
and thus the VC-dimension of finite sets is infinite.
6.9.3 Proof of Main Theorems
We begin with a technical lemma. Consider drawing a set S of n examples from D and
let A denote the event that there exists h ∈ H with zero training error on S but true
error greater than or equal to ɛ. Now draw a second set S ′ of n examples from D and let
B denote the event that there exists h ∈ H with zero error on S but error greater than
or equal to ɛ/2 on S ′ .
Lemma 6.18 Let H be a concept class over some domain X and let S and S ′ be sets of
n elements drawn from some distribution D on X , where n ≥ 8/ɛ. Let A be the event that
there exists h ∈ H with zero error on S but true error greater than or equal to ɛ. Let B
be the event that there exists h ∈ H with zero error on S but error greater than or equal
to ɛ 2 on S′ . Then Prob(B) ≥ Prob(A)/2.
Proof: Clearly, Prob(B) ≥ Prob(A, B) = Prob(A)Prob(B|A). Consider drawing set S
and suppose event A occurs. Let h be in H with err D (h) ≥ ɛ but err S (h) = 0. Now,
draw set S ′ . E(error of h on S ′ ) = err D (h) ≥ ɛ. So, by Chernoff bounds, since n ≥ 8/ɛ,
Prob(err S ′(h) ≥ ɛ/2) ≥ 1/2. Thus, Prob(B|A) ≥ 1/2 and Prob(B) ≥ Prob(A)/2 as
desired.
We now prove Theorem 6.13, restated here for convenience.
211