The error rate of learning halfspaces using kernel-SVM

More documents

Recommendations

Info

$Lectures on fractal geometry and dynamics$

There is also a tight connection between embeddings of S into a Hilbert space and RKHSs.Theorem 5.2 A function k : S × S → R is a kernel iff there exists a mapping φ : S → Hto some real Hilbert space for which k(x, y) = 〈φ(y), φ(x)〉 H . Also,H k = {f v : v ∈ H}Where f v (x) = 〈v, φ(x)〉 H . The mapping v ↦→ f v , restricted to span(φ(S)), is a Hilbert spaceisomorphism.A kernel k : S × S → R is called normalized if sup x∈S k(x, x) = 1. Also,Theorem 5.3 Let k : S × S → R be a kernel and let {f n } ∞ n=1 be an orthonormal basis of aH k . Then, k(x, y) = ∑ ∞n=1 f n(x)f n (y).5.1.2 Unitary Representations of Compact GroupsProofs of the results stated here can be found in (Folland, 1994), chapter 5. Let G be acompact group. A unitary representation (or just a representation) of G is a group homomorphismρ : G → U(H) where U(H) is the class of unitary operators over a Hilbert spaceH, such that, for every v ∈ H, the mapping a ↦→ ρ(a)v is continuous.We say that a closed subspace M ⊂ H is invariant (to ρ) if for every a ∈ G, v ∈ M,ρ(a)v ∈ M. We note that if M is invariant then so is M ⊥ . We denote by ρ| M : G → U(M)the restriction of ρ to M. That is, ∀a ∈ G, ρ| M (a) = ρ(a)| M . We say that ρ : G → U(H) isreducible if H = M ⊕M ⊥ such that M, M ⊥ are both non zero closed and invariant subspacesof H. A basic result is that every representation of a compact group is a sum of irreduciblerepresentation.Theorem 5.4 Let ρ : G → U(H) be a representation of a compact group G. Then, H =⊕ n∈I H n , where every H n is invariant to ρ and ρ| Hn is irreducible.We shall also use the following Lemma.Lemma 5.5 Let G be a compact group, V a finite dimensional vector space and let ρ : G →GL(V ) be a continuous homomorphism of groups (here, GL(V ) is the group of invertiblelinear operators over V ). Then,1. There exists an inner product on V making ρ a unitary representation.2. Moreover, if V has no non-trivial invariant subspaces (here a subspace U ⊂ V is calledinvariant if, ∀a ∈ G, f ∈ U, ρ(a)f ∈ U) then this inner product is unique up to scalarmultiple.17
5.1.3 Harmonic Analysis on the SphereAll the results stated here can be found in (Atkinson and Han, 2012), chapters 1 and2. Denote by O(d) the group of unitary operators over R d and by dA the uniformprobability∫measure over O(d) (that is, dA is the unique probability measure satisfyingf(A)dA = ∫f(AB)dA = ∫f(BA)dA for every B ∈ O(d) and every integrableO(d) O(d) O(d)function f : O(d) → C). Denote by dx = dx d−1 the Lebesgue (area) measure over S d−1 andlet L 2 (S d−1 ) := L 2 (S d−1 , dx). Given a measurable set Z ⊆ S d−1 , we sometime denote itsLebesgue measure by |Z|. Also, denote dm =dx the Lebesgue measure, normalized to be|S d−1 |a probability measure.For every n ∈ N 0 , we denote by Y d n the linear space of d-variables harmonic (i.e., satisfying∆p = 0) homogeneous polynomials of degree n. It holds that( ) ( )d + n − 1 d + n − 3N d,n = dim(Y d n) =−=d − 1 d − 1(2n + d − 2)(n + d − 3)!n!(d − 2)!Denote by P d,n : L 2 (S d−1 ) → Y d n the orthogonal projection onto Y d n.We denote by ρ : O(d) → U(L 2 (S d−1 )) the unitary representation defined byρ(A)f = f ◦ A −1We say that a closed subspace M ⊂ L 2 (S d−1 ) is invariant if it is invariant w.r.t. ρ (that is,∀f ∈ M, A ∈ O(d), f ◦ A ∈ M). We say that an invariant space M is primitive if ρ| M isirreducible.Theorem 5.6 1. L 2 (S d−1 ) = ⊕ ∞ n=0Y d n.2. The primitive finite dimensional subspaces of L 2 (S d−1 ) are exactly {Y d n} ∞ n=0.Lemma 5.7 Fix an orthonormal basis Yn,j, d 1 ≤ j ≤ N d,n to Y d n. For every x ∈ S d−1 itholds thatN d,n∑j=1|Y dn,j(x)| 2 = N d,n|S d−1 |5.1.4 Legendre and Chebyshev PolynomialsThe results stated here can be found at (Atkinson and Han, 2012). Fix d ≥ 2. The ddimensional Legendre polynomials are the sequence of polynomials over [−1, 1] defined bythe recursion formulaP d,n (x) = 2n+d−4xPn+d−3 d,n−1(x) +P d,0 ≡ 1, P d,1 (x) = xn−1n+d−3 P d,n−2(x)We shall make use of the following properties of the Legendre polynomials.Proposition 5.8 1.(For every d ≥ 2, the)sequence {P d,n } is orthogonal basis of theHilbert space L 2 [−1, 1], (1 − x 2 ) d−32 dx .18(8)
Page 1 and 2: The complexity of learning halfspac
Page 3 and 4: exists an equivalent inner product
Page 5 and 6: is enough that we can efficiently c
Page 7 and 8: our terminology, they considered th
Page 9 and 10: It is shown in (Birnbaum and Shalev
Page 11 and 12: We now expand on this brief descrip
Page 13 and 14: and (uniformly and independently) a
Page 15 and 16: The proof of Theorem 2.7To prove Th
Page 17: attempts to prove a quantitative op
Page 21 and 22: Lemma 5.11 (John’s Lemma) (Matous
Page 23 and 24: For 1 ≤ i ≤ t Let v i = x i −
Page 25 and 26: ( )that in this case Err µN ,hinge
Page 27 and 28: Thus, it is enough to find a neighb
Page 29 and 30: Legendre polynomials we have|P d,n
Page 31 and 32: Then, for every K ∈ N, 1 8 > γ >
Page 33 and 34: Now, it holds that∫∫∫ ∫∫
Page 35 and 36: We note that ω f◦g ≤ ω f ·
Page 37 and 38: Now, denote δ = ∫ g. It holds th
Page 39 and 40: equivalent formulationminErr D,l (f
Page 41 and 42: Denote ||g|| Hk = C. By Lemma 5.25,
Page 43 and 44: Consequently, every approximated so
Page 45: Kosaku Yosida. Functional Analysis.

The error rate of learning halfspaces using kernel-SVM

Create successful ePaper yourself

Delete template?

Save as template?