Foundations of Data Science

we argue that if S is large enough compared to some property of H, then with high probability
all h ∈ H have their training error close to their true error, so that if we find a
hypothesis whose training error is low, we can be confident its true error will be low as well.
Before giving our first result of this form, we note that it will often be convenient to
associate each hypotheses with its {−1, 1}-valued indicator function
{ 1 x ∈ h
h(x) =
−1 x ∉ h
In this notation the true error of h is err D (h) = Prob x∼D [h(x) ≠ c ∗ (x)] and the training
error is err S (h) = Prob x∼S [h(x) ≠ c ∗ (x)].
6.2 Overfitting and Uniform Convergence
We now present two results that explain how one can guard against overfitting. Given
a class of hypotheses H, the first result states that for any given ɛ greater than zero, so
long as the training data set is large compared to 1 ln(|H|), it is unlikely any hypothesis
ɛ
h ∈ H will have zero training error but have true error greater than ɛ. This means that
with high probability, any hypothesis that our algorithms finds that agrees with the target
hypothesis on the training data will have low true error. The second result states that if
the training data set is large compared to 1 ln(|H|), then it is unlikely that the training
ɛ 2
error and true error will differ by more than ɛ for any hypothesis in H. This means that if
we find an hypothesis in H whose training error is low, we can be confident its true error
will be low as well, even if its training error is not zero.
The basic idea is the following. If we consider some h with large true error, and we
select an element x ∈ X at random according to D, there is a reasonable chance that
x will belong to the symmetric difference h△c ∗ . If we select a large enough training
sample S with each point drawn independently from X according to D, the chance that
S is completely disjoint from h△c ∗ will be incredibly small. This is just for a single
hypothesis h but we can now apply the union bound over all h ∈ H of large true error,
when H is finite. We formalize this below.
Theorem 6.1 Let H be an hypothesis class and let ɛ and δ be greater than zero. If a
training set S of size
n ≥ 1 ɛ
(
ln |H| + ln(1/δ)
)
,
is drawn from distribution D, then with probability greater than or equal to 1 − δ every h
in H with with true error err D (h) ≥ ɛ has training error err S (h) > 0. Equivalently, with
probability greater than or equal to 1 − δ, every h ∈ H with training error zero has true
error less than ɛ.
Proof: Let h 1 , h 2 , . . . be the hypotheses in H with true error greater than or equal to ɛ.
These are the hypotheses that we don’t want to output. Consider drawing the sample S
192