Foundations of Data Science

discard separators in advance that slice through dense regions and instead focus attention
on just those that indeed separate most of the distribution by a large margin. This is the
high level idea behind a technique known as Semi-Supervised SVMs. Alternatively, suppose
data objects can be described by two different “kinds” of features (e.g., a webpage
could be described using words on the page itself or using words on links pointing to the
page), and one believes that each kind should be sufficient to produce an accurate classifier.
Then one might want to train a pair of classifiers (one on each type of feature) and
use unlabeled data for which one is confident but the other is not to bootstrap, labeling
such examples with the confident classifier and then feeding them as training data to the
less-confident one. This is the high-level idea behind a technique known as Co-Training.
Or, if one believes “similar examples should generally have the same label”, one might
construct a graph with an edge between examples that are sufficiently similar, and aim for
a classifier that is correct on the labeled data and has a small cut value on the unlabeled
data; this is the high-level idea behind graph-based methods.
A formal model: The batch learning model introduced in Sections 6.1 and 6.3 in essence
assumes that one’s prior beliefs about the target function be described in terms of a class
of functions H. In order to capture the reasoning used in semi-supervised learning, we
need to also describe beliefs about the relation between the target function and the data
distribution. A clean way to do this is via a notion of compatibility χ between a hypothesis
h and a distribution D. Formally, χ maps pairs (h, D) to [0, 1] with χ(h, D) = 1
meaning that h is highly compatible with D and χ(h, D) = 0 meaning that h is very
incompatible with D. The quantity 1 − χ(h, D) is called the unlabeled error rate of h, and
denoted err unl (h). Note that for χ to be useful, it must be estimatable from a finite sample;
to this end, let us further require that χ is an expectation over individual examples.
That is, overloading notation for convenience, we require χ(h, D) = E x∼D [χ(h, x)], where
χ : H × X → [0, 1].
For instance, suppose we believe the target should separate most data by margin γ.
We can represent this belief by defining χ(h, x) = 0 if x is within distance γ of the decision
boundary of h, and χ(h, x) = 1 otherwise. In this case, err unl (h) will denote the
probability mass of D within distance γ of h’s decision boundary. As a different example,
in co-training, we assume each example can be described using two “views” that
each are sufficient for classification; that is, there exist c ∗ 1, c ∗ 2 such that for each example
x = 〈x 1 , x 2 〉 we have c ∗ 1(x 1 ) = c ∗ 2(x 2 ). We can represent this belief by defining a hypothesis
h = 〈h 1 , h 2 〉 to be compatible with an example 〈x 1 , x 2 〉 if h 1 (x 1 ) = h 2 (x 2 ) and incompatible
otherwise; err unl (h) is then the probability mass of examples on which h 1 and h 2 disagree.
As with the class H, one can either assume that the target is fully compatible (i.e.,
err unl (c ∗ ) = 0) or instead aim to do well as a function of how compatible the target is.
The case that we assume c ∗ ∈ H and err unl (c ∗ ) = 0 is termed the “doubly realizable
case”. The concept class H and compatibility notion χ are both viewed as known.
229