Foundations of Data Science

√
L log(L(T ))
|S|) will be at most err S (h L ) + O( ), where h
|S|
L = argmin h:L(h)=L err S (h)
is the pruning of T with L leaves of lowest error on S.
3. In the above question, we assumed L was given. Explain √how we can ] remove this assumption
and achieve a bound of min L
[err S (h L ) + O( L log(L(T ))
by instantiating
|S|
)
L(T ) copies of the above algorithm (one for each value of L) and then combining
these algorithms using the experts algorithm (in this case, none of them will be
sleeping).
Exercise 6.9 Kernels; (Section 6.6) Prove Theorem 6.10.
Exercise 6.10 What is the VC-dimension of right corners with axis aligned edges that
are oriented with one edge going to the right and the other edge going up?
Exercise 6.11 (VC-dimension; Section 6.9) What is the VC-dimension V of the
class H of axis-parallel boxes in R d ? That is, H = {h a,b : a, b ∈ R d } where h a,b (x) = 1
if a i ≤ x i ≤ b i for all i = 1, . . . , d and h a,b (x) = −1 otherwise.
1. Prove that the VC-dimension is at least your chosen V by giving a set of V points
that is shattered by the class (and explaining why it is shattered).
2. Prove that the VC-dimension is at most your chosen V by proving that no set of
V + 1 points can be shattered.
Exercise 6.12 (VC-dimension, Perceptron, and Margins; Sections 6.5.3, 6.9)
Say that a set of points S is shattered by linear separators of margin γ if every labeling
of the points in S is achievable by a linear separator of margin at least γ. Prove that no
set of 1/γ 2 + 1 points in the unit ball is shattered by linear separators of margin γ.
Hint: think about the Perceptron algorithm and try a proof by contradiction.
Exercise 6.13 (Linear separators) Suppose the instance space X is {0, 1} d and consider
the target function c ∗ that labels an example x as positive if the least index i for
which x i = 1 is odd, else labels x as negative. In other words, c ∗ (x) = “if x 1 = 1 then
positive else if x 2 = 1 then negative else if x 3 = 1 then positive else ... else negative”.
Show that the rule can be represented by a linear threshold function.
Exercise 6.14 (Linear separators; harder) Prove that for the problem of Exercise
6.13, we cannot have a linear separator with margin at least 1/f(d) where f(d) is bounded
above by a polynomial function of d.
Exercise 6.15 VC-dimension Prove that the VC-dimension of circles in the plane is
three.
Exercise 6.16 VC-dimension Show that the VC-dimension of arbitrary right triangles
in the plane is seven.
235