Machine Learning - DISCo

We define the optimal mistake bound for a concept class C below.
Definition: Let C be an arbitrary nonempty concept class. The optimal mistake
bound for C, denoted Opt (C), is the minimum over all possible learning algorithms
A of MA(C).
Opt (C) = min MA (a
Adearning algorithms
Speaking informally, this definition states that Opt(C) is the number of
mistakes made for the hardest target concept in C, using the hardest training
sequence, by the best algorithm. Littlestone (1987) shows that for any concept
class C, there is an interesting relationship among the optimal mistake bound for
C, the bound of the HALVING algorithm, and the VC dimension of C, namely
Furthermore, there exist concept classes for which the four quantities above
are exactly equal. One such concept class is the powerset Cp of any finite set
of instances X. In this case, VC(Cp) = 1x1 = log2(1CpJ), so all four quantities
must be equal. Littlestone (1987) provides examples of other concept classes for
which VC(C) is strictly less than Opt (C) and for which Opt (C) is strictly less
than M~aIvin~(C)
7.5.4 WEIGHTED-MAJORITY Algorithm
In this section we consider a generalization of the HALVING algorithm called
the WEIGHTED-MAJORITY algorithm. The WEIGHTED-MAJORITY algorithm makes
predictions by taking a weighted vote among a pool of prediction algorithms and
learns by altering the weight associated with each prediction algorithm. These
prediction algorithms can be taken to be the alternative hypotheses in H, or they
can be taken to be alternative learning algorithms that themselves vary over time.
All that we require of a prediction algorithm is that it predict the value of the target
concept, given an instance. One interesting property of the WEIGHTED-MAJORITY
algorithm is that it is able to accommodate inconsistent training data. This is
because it does not eliminate a hypothesis that is found to be inconsistent with
some training example, but rather reduces its weight. A second interesting property
is that we can bound the number of mistakes made by WEIGHTED-MAJORITY in
terms of the number of mistakes committed by the best of the pool of prediction
algorithms.
The WEIGHTED-MAJORITY algorithm begins by assigning a weight of 1 to
each prediction algorithm, then considers the training examples. Whenever a prediction
algorithm misclassifies a new training example its weight is decreased by
multiplying it by some number B, where 0 5 B < 1. The exact definition of the
WEIGHTED-MAJORITY algorithm is given in Table 7.1.
Notice if f? = 0 then WEIGHTED-MAJORITY is identical to the HALVING algorithm.
On the other hand, if we choose some other value for p, no prediction