Support Vector Machines - The Auton Lab

Quadratic Dot
Products
Φ( a)
• Φ(
b)
=
1+
2
m
∑
i=
1
a b +
i
i
m
∑
i=
1
a b
2 2
i i
+
m
m
∑∑
i= 1 j=
i+
1
2a a b b
i
j
i
j
Just out of casual, innocent, interest,
let’s look at another function of a and
b:
( a.
b + 1)
2
= ( a.
b)
+ 2a.
b + 1
2
2
m
m
⎛ ⎞
= ⎜∑
aibi
⎟ + 2∑
aibi
+ 1
⎝ i=
1 ⎠ i=
1
m
m
= ∑∑aibia
jb
j
+ 2∑
aibi
+ 1
i= 1 j=
1
m
m
i=
1
= 2
∑ ( a ) + 2∑∑
+ 2 ∑
ibi
aibi
a
jb
j
aibi
+ 1
i=
1
m
m
i= 1 j=
i+
1
m
i=
1
They’re the same!
And this is only O(m) to
compute!
Copyright © 2001, 2003, Andrew W. Moore Support Vector Machines: Slide 51
Maximize
QP with Quadratic basis functions
R
R R
1
∑αk
− ∑∑αkαlQkl
Qkl
= yk
yl
Φ(
xk
).
k = 1 2 k = 1 l=
1
Subject to these
constraints:
0 ≤
α k
≤ C
where ( Φ(
xl
))
We must do R 2 /2 dot products to
get this matrix ready. R
∀k
Warning: up until Rong Zhang spotted my error in
Oct 2003, this equation had been wrong in earlier
versions of the notes. This version is correct.
∑
α k
y k
Each dot product now only requires
m additions and multiplications
k = 1
=
0
Then define:
w =
b
k s.t.
= y (1 − ε ) − x . w
K
∑
α k
α
y k k
> 0
where K = arg max
K
Φ ( x
k
K
k
)
α
K
k
Then classify with:
f(x,w,b) = sign(w. φ(x) -b)
Copyright © 2001, 2003, Andrew W. Moore Support Vector Machines: Slide 52
26