Foundations of Data Science

Not spam
{ }} { {
Spam
}} {
x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x 11 x 12 x 13 x 14 x 15 x 16 emails
↓ ↓ ↓
0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 target concept
↕ ↕ ↕
0 1 0 0 0 0 1 0 1 1 1 0 1 0 1 1 hypothesis h i
↑ ↑ ↑
Figure 6.1: The hypothesis h i disagrees with the truth in one quarter of the emails. Thus
with a training set |S|, the probability that the hypothesis will survive is (1 − 0.25) |S|
of size n and let A i be the event that h i is consistent with S. Since every h i has true error
greater than or equal to ɛ
Prob(A i ) ≤ (1 − ɛ) n .
In other words, if we fix h i and draw a sample S of size n, the chance that h i makes no
mistakes on S is at most the probability that a coin of bias ɛ comes up tails n times in a
row, which is (1 − ɛ) n . By the union bound over all i we have
Prob (∪ i A i ) ≤ |H|(1 − ɛ) n .
Using the fact that (1−ɛ) ≤ e −ɛ , the probability that any hypothesis in H with true error
greater than or equal to ɛ has training error zero is at most |H|e −ɛn . Replacing n by the
sample size bound from the theorem statement, this is at most |H|e − ln |H|−ln(1/δ) = δ as
desired.
The conclusion of Theorem 6.1 is sometimes called a “PAC-learning guarantee” since
it states that if we can find an h ∈ H consistent with the sample, then this h is Probably
Approximately Correct.
Theorem 6.1 addressed the case where there exists a hypothesis in H with zero training
error. What if the best h i in H has 5% error on S? Can we still be confident that its
true error is low, say at most 10%? For this, we want an analog of Theorem 6.1 that says
for a sufficiently large training set S, every h i ∈ H has training error within ±ɛ of the
true error with high probability. Such a statement is called uniform convergence because
we are asking that the training set errors converge to their true errors uniformly over all
sets in H. To see intuitively why such a statement should be true for sufficiently large
S and a single hypothesis h i , consider two strings that differ in 10% of the positions and
randomly select a large sample of positions. The number of positions that differ in the
sample will be close to 10%.
To prove uniform convergence bounds, we use a tail inequality for sums of independent
Bernoulli random variables (i.e., coin tosses). The following is particularly convenient and
is a variation on the Chernoff bounds in Section 12.4.11 of the appendix.
193