Foundations of Data Science

6 Machine Learning
6.1 Introduction
Machine learning algorithms are general purpose tools that solve problems from many
disciplines without detailed domain-specific knowledge. They have proven to be very
effective in a large number of contexts, including computer vision, speech recognition,
document classification, automated driving, computational science, and decision support.
The core problem. A core problem underlying many machine learning applications
is learning a good classification rule from labeled data. This problem consists of a domain
of interest X , called the instance space, such as email messages or patient records,
and a classification task, such as classifying email messages into spam versus non-spam
or determining which patients will respond well to a given medical treatment. We will
typically assume our instance space X = {0, 1} d or X = R d , corresponding to data that is
described by d Boolean or real-valued features. Features for email messages could be the
presence or absence of various types of words, and features for patient records could be
the results of various medical tests. To perform the learning task, our learning algorithm
is given a set S of labeled training examples, which are points in X along with their
correct classification. This training data could be a collection of email messages, each
labeled as spam or not spam, or a collection of patients, each labeled by whether or not
they responded well to the given medical treatment. Our algorithm then aims to use the
training examples to produce a classification rule that will perform well over new data.
A key feature of machine learning, which distinguishes it from other algorithmic tasks, is
that our goal is generalization: to use one set of data in order to perform well on new data
we have not seen yet. We focus on binary classification where items in the domain of interest
are classified into two categories, as in the medical and spam-detection examples above.
How to learn. A high-level approach to solving this problem that many algorithms
we discuss will follow is to try to find a “simple” rule with good performance on the
training data. For instance in the case of classifying email messages, we might find a set
of highly indicative words such that every spam email in the training data has at least
one of these words and none of the non-spam emails has any of them; in this case, the
rule “if the message has any of these words then it is spam, else it is not” would be a
simple rule that performs well on the training data. Or, we might find a way of weighting
words with positive and negative weights such that the total weighted sum of words in
the email message is positive on the spam emails in the training data, and negative on the
non-spam emails. We will then argue that so long as the training data is representative of
what future data will look like, we can be confident that any sufficiently “simple” rule that
performs well on the training data will also perform well on future data. To make this into
a formal mathematical statement, we need to be precise about what we mean by “simple”
as well as what it means for training data to be “representative” of future data. In fact,
we will see several notions of complexity, including bit-counting and VC-dimension, that
190