Foundations of Data Science

Notice that the sum of slack variables is the total hinge loss of w. So, this convex
optimization is minimizing a weighted sum of 1/γ 2 , where γ is the margin, and the total
hinge loss. If we were to add the constraint that all s i = 0 then this would be solving for
the maximum margin linear separator for the data. However, in practice, optimizing a
weighted combination generally performs better.
6.9 VC-Dimension
In Section 6.2 we presented several theorems showing that so long as the training set
S is large compared to 1 ɛ log(|H|), we can be confident that every h ∈ H with err D(h) ≥ ɛ
will have err S (h) > 0, and if S is large compared to 1 ɛ 2 log(|H|), then we can be confident
that every h ∈ H will have |err D (h)−err S (h)| ≤ ɛ. In essence, these results used log(|H|)
as a measure of complexity of class H. VC-dimension is a different, tighter measure of
complexity for a concept class, and as we will see, is also sufficient to yield confidence
bounds. For any class H, VCdim(H) ≤ log 2 (|H|) but it can also be quite a bit smaller.
Let’s introduce and motivate it through an example.
Consider a database consisting of the salary and age for a random sample of the adult
population in the United States. Suppose we are interested in using the database to answer
questions of the form: “what fraction of the adult population in the United States
has age between 35 and 45 and salary between $50,000 and $70,000?” That is, we are
interested in queries that ask about the fraction of the adult population within some axisparallel
rectangle. What we can do is calculate the fraction of the database satisfying
this condition and return this as our answer. This brings up the following question: How
large does our database need to be so that with probability greater than or equal to 1−δ,
our answer will be within ±ɛ of the truth for every possible rectangle query of this form?
If we assume our values are discretized such as 100 possible ages and 1,000 possible
salaries, then there are at most (100 × 1, 000) 2 = 10 10 possible rectangles. This means we
can apply Theorem 6.3 with |H| ≤ 10 10 . Specifically, we can think of the target concept
c ∗ as the empty set so that err S (h) is exactly the fraction of the sample inside rectangle
h and err D (h) is exactly the fraction of the whole population inside h. 22 This would tell
us that a sample size of
1
2ɛ 2 (10 ln 10 + ln(2/δ)) would be sufficient.
However, what if we do not wish to discretize our concept class? Another approach
would be to say that if there are only N adults total in the United States, then there
are at most N 4 rectangles that are truly different with respect to D and so we could use
|H| ≤ N 4 . Still, this suggests that S needs to grow with N, albeit logarithmically, and
one might wonder if that is really necessary. VC-dimension, and the notion of the growth
22 Technically D is the uniform distribution over the adult population of the United States, and we
want to think of S as an independent identical distributed sample from this D.
206