Foundations of Data Science

will allow us to make mathematical statements of this form. These statements can be
viewed as formalizing the intuitive philosophical notion of Occam’s razor.
Formalizing the problem. To formalize the learning problem, assume there is some
probability distribution D over the instance space X , such that (a) our training set S
consists of points drawn independently at random from D, and (b) our objective is to
predict well on new points that are also drawn from D. This is the sense in which we
assume that our training data is representative of future data. Let c ∗ , called the target
concept, denote the subset of X corresponding to the positive class for the binary classification
we are aiming to make. For example, c ∗ would correspond to the set of all patients
who respond well to the treatment in the medical example, or the set of all spam emails
in the spam-detection setting. So, each point in our training set S is labeled according
to whether or not it belongs to c ∗ and our goal is to produce a set h ⊆ X , called our
hypothesis, which is close to c ∗ with respect to distribution D. The true error of h is
err D (h) = Prob(h△c ∗ ) where “△” denotes symmetric difference, and probability mass is
according to D. In other words, the true error of h is the probability it incorrectly classifies
a data point drawn at random from D. Our goal is to produce h of low true error.
The training error of h, denoted err S (h), is the fraction of points in S on which h and
c ∗ disagree. That is, err S (h) = |S ∩ (h△c ∗ )|/|S|. Training error is also called empirical
error. Note that even though S is assumed to consist of points randomly drawn from D,
it is possible for a hypothesis h to have low training error or even to completely agree with
c ∗ over the training sample, and yet have high true error. This is called overfitting the
training data. For instance, a hypothesis h that simply consists of listing the positive examples
in S, which is equivalent to a rule that memorizes the training sample and predicts
positive on an example if and only if it already appeared positively in the training sample,
would have zero training error. However, this hypothesis likely would have high true error
and therefore would be highly overfitting the training data. More generally, overfitting is
a concern because algorithms will typically be optimizing over the training sample. To
design and analyze algorithms for learning, we will have to address the issue of overfitting.
To be able to formally analyze overfitting, we introduce the notion of an hypothesis
class, also called a concept class or set system. An hypothesis class H over X is a collection
of subsets of X , called hypotheses. For instance, the class of intervals over X = R is the
collection {[a, b]|a ≤ b}. The class of linear separators over X = R d is the collection
{{x ∈ R d |w · x ≥ w 0 }|w ∈ R d , w 0 ∈ R};
that is, it is the collection of all sets in R d that are linearly separable from their complement.
In the case that X is the set of 4 points in the plane {(−1, −1), (−1, 1), (1, −1), (1, 1)},
the class of linear separators contains 14 of the 2 4 = 16 possible subsets of X . 17 Given an
hypothesis class H and training set S, what we typically aim to do algorithmically is to
find the hypothesis in H that most closely agrees with c ∗ over S. To address overfitting,
17 The only two subsets that are not in the class are the sets {(−1, −1), (1, 1)} and {(−1, 1), (1, −1)}.
191