Foundations of Data Science

6.16 Exercises
Exercise 6.1 (Section 6.2 and 6.3) Consider the instance space X = {0, 1} d and let
H be the class of 3-CNF formulas. That is, H is the set of concepts that can be described
as a conjunction of clauses where each clause is an OR of up to 3 literals. (These are also
called 3-SAT formulas). For example c ∗ might be (x 1 ∨¯x 2 ∨x 3 )(x 2 ∨x 4 )(¯x 1 ∨x 3 )(x 2 ∨x 3 ∨x 4 ).
Assume we are in the PAC learning setting, so examples are drawn from some underlying
distribution D and labeled by some 3-CNF formula c ∗ .
1. Give a number of samples m that would be sufficient to ensure that with probability
≥ 1 − δ, all 3-CNF formulas consistent with the sample have error at most ɛ with
respect to D.
2. Give a polynomial-time algorithm for PAC-learning the class of 3-CNF formulas.
Exercise 6.2 (Section 6.2) Consider the instance space X = R, and the class of functions
H = {f a : f a (x) = 1 iff x ≥ a} for a ∈ R. That is, H is the set of all threshold
functions on the line. Prove that for any distribution D, a sample S of size O( 1 log( 1))
ɛ δ
is sufficient to ensure that with probability ≥ 1 − δ, any f a ′ such that err S (f a ′) = 0 has
err D (f a ′) ≤ ɛ. Note that you can answer this question from first principles, without using
the concept of VC-dimension.
Exercise 6.3 (Perceptron; Section 6.5.3) Consider running the Perceptron algorithm
in the online model on some sequence of examples S. Let S ′ be the same set of examples
as S but presented in a different order. Does the Perceptron algorithm necessarily make
the same number of mistakes on S as it does on S ′ ? If so, why? If not, show such an S
and S ′ (consisting of the same set of examples in a different order) where the Perceptron
algorithm makes a different number of mistakes on S ′ than it does on S.
Exercise 6.4 (representation and linear separators) Show that any disjunction (see
Section 6.3.1) over {0, 1} d can be represented as a linear separator. Show that moreover
the margin of separation is Ω(1/ √ d).
Exercise 6.5 (Linear separators; easy) Show that the parity function on d ≥ 2
Boolean variables cannot be represented by a linear threshold function. The parity function
is 1 if and only if an odd number of inputs is 1.
Exercise 6.6 (Perceptron; Section 6.5.3) We know the Perceptron algorithm makes
at most 1/γ 2 mistakes on any sequence of examples that is separable by margin γ (we
assume all examples are normalized to have length 1). However, it need not find a separator
of large margin. If we also want to find a separator of large margin, a natural
alternative is to update on any example x such that f ∗ (x)(w · x) < 1; this is called the
margin perceptron algorithm.
1. Argue why margin perceptron is equivalent to running stochastic gradient descent on
the class of linear predictors (f w (x) = w · x) using hinge loss as the loss function
and using λ t = 1.
233