Machine Learning - DISCo

CHAPTER 7 COMPUTATIONAL LEARNING THEORY 211
probability of the coin being heads corresponds to the probability that the hypothesis
will misclassify a randomly drawn instance. The m independent coin flips correspond
to the m independently drawn instances. The frequency of heads over the m
examples corresponds to the frequency of misclassifications over the m instances.
The Hoeffding bounds state that if the training error errOrD(h) is measured
over the set D containing m randomly drawn examples, then
This gives us a bound on the probability that an arbitrarily chosen single hypothesis
has a very misleading training error. To assure that the best hypothesis found by
L has an error bounded in this way, we must consider the probability that any
one of the 1 H 1 hypotheses could have a large error
Pr[(3h E H)(errorv(h) > error~(h) + E)] 5 1 H ~ e - ~ ~ ' ~
If we call this probability 6, and ask how many examples m suffice to hold S to
some desired value, we now obtain
This is the generalization of Equation (7.2) to the case in which the learner still
picks the best hypothesis h E H, but where the best hypothesis may have nonzero
training error. Notice that m depends logarithmically on H and on 116, as it did
in the more restrictive case of Equation (7.2). However, in this less restrictive
situation m now grows as the square of 116, rather than linearly with 116.
7.3.2 Conjunctions of Boolean Literals Are PAC-Learnable
Now that we have a bound indicating the number of training examples sufficient
to probably approximately learn the target concept, we can use it to determine the
sample complexity and PAC-learnability of some specific concept classes.
Consider the class C of target concepts described by conjunctions of boolean
literals. A boolean literal is any boolean variable (e.g., Old), or its negation (e.g.,
-Old). Thus, conjunctions of boolean literals include target concepts such as
"Old A -Tallv. Is C PAC-learnable? We can show that the answer is yes by
first showing that any consistent learner will require only a polynomial number
of training examples to learn any c in C, and then suggesting a specific algorithm
that uses polynomial time per training example.
Consider any consistent learner L using a hypothesis space H identical to C.
We can use Equation (7.2) to compute the number m of random training examples
sufficient to ensure that L will, with probability (1 - S), output a hypothesis with
maximum error E. To accomplish this, we need only determine the size IHI of
the hypothesis space.
Now consider the hypothesis space H defined by conjunctions of literals
based on n boolean variables. The size 1HI of this hypothesis space is 3". To see
this, consider the fact that there are only three possibilities for each variable in