Foundations of Data Science

Theorem 6.2 (Hoeffding bounds) Let x 1 , x 2 , . . . , x n be independent {0, 1}-valued random
variables with probability p that x i equals one. Let s = ∑ i x i (equivalently, flip n
coins of bias p and let s be the total number of heads). For any 0 ≤ α ≤ 1,
Prob(s/n > p + α) ≤ e −2nα2
Prob(s/n < p − α) ≤ e −2nα2 .
Theorem 6.2 implies the following uniform convergence analog of Theorem 6.1.
Theorem 6.3 (Uniform convergence) Let H be a hypothesis class and let ɛ and δ be
greater than zero. If a training set S of size
n ≥ 1
2ɛ 2 (
ln |H| + ln(2/δ)
)
,
is drawn from distribution D, then with probability greater than or equal to 1 − δ, every h
in H satisfies |err S (h) − err D (h)| ≤ ɛ.
Proof: First, fix some h ∈ H and let x j be the indicator random variable for the event
that h makes a mistake on the j th example in S. The x j are independent {0, 1} random
variables and the probability that x i equals 1 is the true error of h, and the fraction of the
x j ’s equal to 1 is exactly the training error of h. Therefore, Hoeffding bounds guarantee
that the probability of the event A h that |err D (h) − err S (h)| > ɛ is less than or equal to
2e −2nɛ2 . Applying the union bound to the events A h over all h ∈ H, the probability that
there exists an h ∈ H with the difference between true error and empirical error greater
than ɛ is less than or equal to 2|H|e −2nɛ2 . Using the value of n from the theorem statement,
the right-hand-side of the above inequality is at most δ as desired.
Theorem 6.3 justifies the approach of optimizing over our training sample S even if we
are not able to find a rule of zero training error. If our training set S is sufficiently large,
with high probability, good performance on S will translate to good performance on D.
Note that Theorems 6.1 and 6.3 require |H| to be finite in order to be meaningful.
The notion of growth functions and VC-dimension in Section 6.9, extend Theorem 6.3 to
certain infinite hypothesis classes.
6.3 Illustrative Examples and Occam’s Razor
We now present some examples to illustrate the use of Theorem 6.1 and 6.3 and also
use these theorems to give a formal connection to the notion of Occam’s razor.
6.3.1 Learning disjunctions
Consider the instance space X = {0, 1} d and suppose we believe that the target concept
can be represented by a disjunction (an OR) over features, such as c ∗ = {x|x 1 = 1 ∨ x 4 =
194