Support Vector Machines - The Auton Lab

Higher Order Polynomials
Polynomial
Quadratic
Cubic
Quartic
φ(x)
All m 2 /2
terms up to
degree 2
All m 3 /6
terms up to
degree 3
All m 4 /24
terms up to
degree 4
Cost to
build Q kl
matrix
tradition
ally
m 2 R 2 /4
Cost if 100
inputs
2,500 R 2
m 3 R 2 /12 83,000 R 2
m 4 R 2 /48 1,960,000 R 2
Cost to
build Q kl
matrix
sneakily
φ(a).φ(b)
(a.b+1) 2
(a.b+1) 3
(a.b+1) 4
mR 2 / 2
mR 2 / 2
mR 2 / 2
Cost if
100
inputs
50 R 2
50 R 2
50 R 2
Copyright © 2001, 2003, Andrew W. Moore Support Vector Machines: Slide 53
QP with Quintic basis functions
We must do R 2 /2 dot products R R to get this
matrix ready.
Maximize ∑αk
+ ∑∑αkαlQ
where Q ( ( ). ( ))
kl
kl
= yk
yl
Φ xk
Φ xl
In 100-d, each k = 1dot product k = 1now l=
1 needs 103
operations instead of 75 million
R
But there are still worrying things lurking away.
Subject to these
What are they? 0 ≤ α k
≤ C ∀k
∑ α k
y k
= 0
constraints:
k = 1
Then define:
w =
b
k s.t.
K
∑
α k
α
y k k
> 0
K
Φ ( x
= y (1 − ε ) − x . w
where K = arg max
k
K
k
)
α
K
k
Then classify with:
f(x,w,b) = sign(w. φ(x) -b)
Copyright © 2001, 2003, Andrew W. Moore Support Vector Machines: Slide 54
27