Foundations of Data Science

Define a half space to be the set of all points on one side of a hyper plane, i.e., a set
of the form {x|w T x ≥ w 0 }. The VC-dimension of half spaces in d-dimensions is d + 1.
There exists a set of size d + 1 that can be shattered by half spaces. Select the d
unit-coordinate vectors plus the origin to be the d + 1 points. Suppose A is any subset
of these d + 1 points. Without loss of generality assume that the origin is in A. Take
a 0-1 vector w which has 1’s precisely in the coordinates corresponding to vectors not
in A. Clearly A lies in the half-space w T x ≤ 0 and the complement of A lies in the
complementary half-space.
We now show that no set of d + 2 points in d-dimensions can be shattered by linear
separators. This is done by proving that any set of d+2 points can be partitioned into two
disjoint subsets A and B of points whose convex hulls intersect. This establishes the claim
since any linear separator with A on one side must have its entire convex hull on that
side, 23 so it is not possible to have a linear separator with A on one side and B on the other.
Let convex(S) denote the convex hull of point set S.
Theorem 6.17 (Radon): Any set S ⊆ R d with |S| ≥ d + 2, can be partitioned into two
disjoint subsets A and B such that convex(A) ∩ convex(B) ≠ φ.
Proof: Without loss of generality, assume |S| = d+2. Form a d×(d+2) matrix with one
column for each point of S. Call the matrix A. Add an extra row of all 1’s to construct a
(d+1)×(d+2) matrix B. Clearly the rank of this matrix is at most d+1 and the columns
are linearly dependent. Say x = (x 1 , x 2 , . . . , x d+2 ) is a nonzero vector with Bx = 0.
Reorder the columns so that x 1 , x 2 , . . . , x s ≥ 0 and x s+1 , x s+2 , . . . , x d+2 < 0. Normalize
x so
s ∑
i=1
|x i | = 1. Let b i (respectively a i ) be the i th column of B (respectively A). Then,
s∑
|x i |b i = d+2 ∑
i=1
d+2
∑
i=s+1
i=s+1
|x i |. Since s ∑
i=1
|x i |b i from which it follows that
|x i | = 1 and d+2 ∑
i=s+1
s∑
i=1
|x i | = 1 each side of
|x i |a i = d+2 ∑
i=s+1
s∑
|x i |a i = d+2 ∑
i=1
∑
|x i |a i and s |x i | =
i=s+1
i=1
|x i |a i is a convex
combination of columns of A which proves the theorem. Thus, S can be partitioned into
two sets, the first consisting of the first s points after the rearrangement and the second
consisting of points s + 1 through d + 2 . Their convex hulls intersect as required.
Radon’s theorem immediately implies that half-spaces in d-dimensions do not shatter
any set of d + 2 points.
Spheres in d-dimensions
23 If any two points x 1 and x 2 lie on the same side of a separator, so must any convex combination: if
w · x 1 ≥ b and w · x 2 ≥ b then w · (ax 1 + (1 − a)x 2 ) ≥ b.
210