Foundations of Data Science

Intuition: In this framework, the way that unlabeled data helps in learning can be intuitively
described as follows. Suppose one is given a concept class H (such as linear
separators) and a compatibility notion χ (such as penalizing h for points within distance
γ of the decision boundary). Suppose also that one believes c ∗ ∈ H (or at least is close)
and that err unl (c ∗ ) = 0 (or at least is small). Then, unlabeled data can help by allowing
one to estimate the unlabeled error rate of all h ∈ H, thereby in principle reducing the
search space from H (all linear separators) down to just the subset of H that is highly
compatible with D. The key challenge is how this can be done efficiently (in theory,
in practice, or both) for natural notions of compatibility, as well as identifying types of
compatibility that data in important problems can be expected to satisfy.
A theorem: The following is a semi-supervised analog of our basic sample complexity
theorem, Theorem 6.1. First, fix some set of functions H and compatibility notion χ.
Given a labeled sample L, define êrr(h) to be the fraction of mistakes of h on L. Given
an unlabeled sample U, define χ(h, U) = E x∼U [χ(h, x)] and define êrr unl (h) = 1−χ(h, U).
That is, êrr(h) and êrr unl (h) are the empirical error rate and unlabeled error rate of h,
respectively. Finally, given α > 0, define H D,χ (α) to be the set of functions f ∈ H such
that err unl (f) ≤ α.
Theorem 6.22 If c ∗ ∈ H then with probability at least 1 − δ, for labeled set L and
unlabeled set U drawn from D, the h ∈ H that optimizes êrr unl (h) subject to êrr(h) = 0
will have err D (h) ≤ ɛ for
|U| ≥ 2 [
ln |H| + ln 4 ]
, and |L| ≥ 1 [
ln |H
ɛ 2 D,χ (err unl (c ∗ ) + 2ɛ)| + ln 2 ]
.
δ
ɛ
δ
Equivalently, for |U| satisfying this bound, for any |L|, whp the h ∈ H that minimizes
êrr unl (h) subject to êrr(h) = 0 has
err D (h) ≤ 1 [
ln |H D,χ (err unl (c ∗ ) + 2ɛ)| + ln 2 ]
.
|L|
δ
Proof: By Hoeffding bounds, |U| is sufficiently large so that with probability at least
1 − δ/2, all h ∈ H have |êrr unl (h) − err unl (h)| ≤ ɛ. Thus we have:
{f ∈ H : êrr unl (f) ≤ err unl (c ∗ ) + ɛ} ⊆ H D,χ (err unl (c ∗ ) + 2ɛ).
The given bound on |L| is sufficient so that with probability at least 1 − δ, all h ∈ H with
êrr(h) = 0 and êrr unl (h) ≤ err unl (c ∗ ) + ɛ have err D (h) ≤ ɛ; furthermore, êrr unl (c ∗ ) ≤
err unl (c ∗ ) + ɛ, so such a function h exists. Therefore, with probability at least 1 − δ, the
h ∈ H that optimizes êrr unl (h) subject to êrr(h) = 0 has err D (h) ≤ ɛ, as desired.
One can view Theorem 6.22 as bounding the number of labeled examples needed to learn
well as a function of the “helpfulness” of the distribution D with respect to χ. Namely,
a helpful distribution is one in which H D,χ (α) is small for α slightly larger than the
compatibility of the true target function, so we do not need much labeled data to identify a
good function among those in H D,χ (α). For more information on semi-supervised learning,
see [?, ?, ?, ?, ?].
230