Machine Learning - DISCo

C that contains every teachable concept relative to X. The set C of all definable
target concepts corresponds to the power set of X-the set of all subsets of X-
which contains ICI = 2IXI concepts. Suppose that instances in X are defined by
n boolean features. In this case, there will be 1x1 = 2" distinct instances, and
therefore ICI = 21'1 = 2' distinct concepts. Of course to learn such an unbiased
concept class, the learner must itself use an unbiased hypothesis space H = C.
Substituting I H I = 22n into Equation (7.2) gives the sample complexity for learning
the unbiased concept class relative to X.
Thus, this unbiased class of target concepts has exponential sample complexity
under the PAC model, according to Equation (7.2). Although Equations (7.2)
and (7.5) are not tight upper bounds, it can in fact be proven that the sample
complexity for the unbiased concept class is exponential in n.
I 7.3.3.2 K-TERM DNF AND K-CNF CONCEPTS
I1
It is also possible to find concept classes that have polynomial sample complexity,
but nevertheless cannot be learned in polynomial time. One interesting example is
the concept class C of k-term disjunctive normal form (k-term DNF) expressions.
k-term DNF expressions are of the form TI v T2 v . . - v Tk, where each term 1;:
is a conjunction of n boolean attributes and their negations. Assuming H = C, it
is easy to show that I HI is at most 3"k (because there are k terms, each of which
may take on 3" possible values). Note 3"k is an overestimate of H, because it is
double counting the cases where = I;. and where 1;: is more_general-than I;..
Still, we can use this upper bound on I HI to obtain an upper bound on the sample
complexity, substituting this into Equation (7.2).
which indicates that the sample complexity of k-term DNF is polynomial in
1/~, 116, n, and k. Despite having polynomial sample complexity, the computational
complexity is not polynomial, because this learning problem can be shown
to be equivalent to other problems that are known to be unsolvable in polynomial
time (unless RP = NP). Thus, although k-term DNF has polynomial sample
complexity, it does not have polynomial computational complexity for a learner
using H = C.
The surprising fact about k-term DNF is that although it is not PAClearnable,
there is a strictly larger concept class that is! This is possible because
the larger concept class has polynomial computation complexity per example and
still has polynomial sample complexity. This larger class is the class of k-CNF
expressions: conjunctions of arbitrary length of the form TI A T2 A. .. A I;., where
each is a disjunction of up to k boolean attributes. It is straightforward to show
that k-CNF subsumes k-DNF, because any k-term DNF expression can easily be