Foundations of Data Science

containing point x, define twin(h) = h \ {x}; this may or may not belong to H[S]. Let
T = {h ∈ H[S] : x ∈ h and twin(h) ∈ H[S]}. Notice |H[S]| − |H[S \ {x}]| = |T |.
Now, what is the VC-dimension of T ? If d ′ = VCdim(T ), this means there is some set
R of d ′ points in S \ {x} that are shattered by T . By definition of T , all 2 d′ subsets of R
can be extended to either include x, or not include x and still be a set in H[S]. In other
words, R ∪ {x} is shattered by H. This means, d ′ + 1 ≤ d. Since VCdim(T ) ≤ d − 1, by
induction we have |T | ≤ ( n−1
≤d−1)
as desired.
6.9.4 VC-dimension of combinations of concepts
Often one wants to create concepts out of other concepts. For example, given several
linear separators, one could take their intersection to create a convex polytope. Or given
several disjunctions, one might want to take their majority vote. We can use Sauer’s
lemma to show that such combinations do not increase the VC-dimension of the class by
too much.
Specifically, given k concepts h 1 , h 2 , . . . , h k and a Booelan function f define the set
comb f (h 1 , . . . , h k ) = {x ∈ X : f(h 1 (x), . . . , h k (x)) = 1}, where here we are using h i (x) to
denote the indicator for whether or not x ∈ h i . For example, f might be the AND function
to take the intersection of the sets h i , or f might be the majority-vote function. This can
be viewed as a depth-two neural network. Given a concept class H, a Boolean function f,
and an integer k, define the new concept class COMB f,k (H) = {comb f (h 1 , . . . , h k ) : h i ∈
H}. We can now use Sauer’s lemma to produce the following corollary.
Corollary 6.19 If the concept class H has VC-dimension d, then for any combination
function f, the class COMB f,k (H) has VC-dimension O ( kd log(kd) ) .
Proof: Let n be the VC-dimension of COMB f,k (H), so by definition, there must exist
a set S of n points shattered by COMB f,k (H). We know by Sauer’s lemma that there
are at most n d ways of partitioning the points in S using sets in H. Since each set in
COMB f,k (H) is determined by k sets in H, and there are at most (n d ) k = n kd different
k-tuples of such sets, this means there are at most n kd ways of partitioning the points
using sets in COMB f,k (H). Since S is shattered, we must have 2 n ≤ n kd , or equivalently
n ≤ kd log 2 (n). We solve this as follows. First, assuming n ≥ 16 we have log 2 (n) ≤ √ n so
kd log 2 (n) ≤ kd √ n which implies that n ≤ (kd) 2 . To get the better bound, plug back into
the original inequality. Since n ≤ (kd) 2 , it must be that log 2 (n) ≤ 2 log 2 (kd). substituting
log n ≤ 2 log 2 (kd) into n ≤ kd log 2 n gives n ≤ 2kd log 2 (kd).
This result will be useful for our discussion of Boosting in Section 6.10.
6.9.5 Other measures of complexity
VC-dimension and number of bits needed to describe a set are not the only measures
of complexity one can use to derive generalization guarantees. There has been significant
214