Machine Learning - DISCo

est possible hypothesis in H, after observing rn randomly drawn training
examples, provided
a The number of training examples required for successful learning is strongly
influenced by the complexity of the hypothesis space considered by the
learner. One useful measure of the complexity of a hypothesis space H
is its Vapnik-Chervonenkis dimension, VC(H). VC(H) is the size of the
largest subset of instances that can be shattered (split in all possible ways)
by H.
a An alternative upper bound on the number of training examples sufficient
for successful learning under the PAC model, stated in terms of VC(H) is
A lower bound is
a An alternative learning model, called the mistake bound model, is used to
analyze the number of training examples a learner will misclassify before
it exactly learns the target concept. For example, the HALVING algorithm
will make at most Llog, 1 H 1 J mistakes before exactly learning any target
concept drawn from H. For an arbitrary concept class C, the best worstcase
algorithm will make Opt (C) mistakes, where
VC(C> 5 Opt(C) I log,(lCI)
a The WEIGHTED-MAJORITY algorithm combines the weighted votes of multiple
prediction algorithms to classify new instances. It learns weights for each of
these prediction algorithms based on errors made over a sequence of examples.
Interestingly, the number of mistakes made by WEIGHTED-MAJORITY can
be bounded in terms of the number of mistakes made by the best prediction
algorithm in the pool.
Much early work on computational learning theory dealt with the question
of whether the learner could identify the target concept in the limit, given an
indefinitely long sequence of training examples. The identification in the limit
model was introduced by Gold (1967). A good overview of results in this area is
(Angluin 1992). Vapnik (1982) examines in detail the problem of uniform convergence,
and the closely related PAC-learning model was introduced by Valiant
(1984). The discussion in this chapter of €-exhausting the version space is based
on Haussler's (1988) exposition. A useful collection of results under the PAC
model can be found in Blumer et al. (1989). Kearns and Vazirani (1994) provide
an excellent exposition of many results from computational learning theory.
Earlier texts in this area include Anthony and Biggs (1992) and Natarajan (1991).