Support Vector Machines - The Auton Lab

Maximize
QP with Quintic basis functions
We must do R 2 /2 dot products R R to get this
matrix ready.
∑αk
+ ∑∑αkαlQ
where Q ( ( ). ( ))
kl
kl
= yk
yl
Φ xk
Φ xl
In 100-d, each k = 1dot product k = 1now l=
1 needs 103
The use of Maximum Margin
operations instead of 75 million
magically makes this not a
problem R
But there are still worrying things lurking away.
Subject to these
What are they? 0 ≤ α k
≤ C ∀k
∑ α k
y k
= 0
constraints:
•The fear of overfitting kwith = 1 this enormous
number of terms
Then define:
•The evaluation phase (doing a set of
predictions on a test set) will be very
expensive (why?)
w
=
k s.t.
∑
α k
α
y k k
> 0
Φ ( x
wb
⋅= Φ ( yx
) =
K
(1 −∑
εα
k
y
K
) k
Φ−
(
x
k
)
K
. ⋅wΦ
k s.t. α k > 0
where
=
K∑=
α
arg k
y k
( xmax
k
⋅ x + 1)
α
k s.t. α k > 0
k
k
Only Sm operations (S=#support vectors)
k
)
( x )
K
5
Because each w. φ(x) (see below)
needs 75 million operations. What
can be done?
Then classify with:
f(x,w,b) = sign(w. φ(x) -b)
Copyright © 2001, 2003, Andrew W. Moore Support Vector Machines: Slide 57
Maximize
QP with Quintic basis functions
We must do R 2 /2 dot products R R to get this
matrix ready.
∑αk
+ ∑∑αkαlQ
where Q ( ( ). ( ))
kl
kl
= yk
yl
Φ xk
Φ xl
In 100-d, each k = 1dot product k = 1now l=
1 needs 103
The use of Maximum Margin
operations instead of 75 million
magically makes this not a
problem R
But there are still worrying things lurking away.
Subject to these
What are they? 0 ≤ α k
≤ C ∀k
∑ α k
y k
= 0
constraints:
•The fear of overfitting kwith = 1 this enormous
number of terms
Then define:
•The evaluation phase (doing a set of
predictions on a test set) will be very
expensive (why?)
wb
w
⋅
=
k s.t.
∑
α k
α
y k k
> 0
Φ ( x
= Φ ( yx
) = (1 −∑
εα
) Φ−
(
x ) . ⋅wΦ
k
)
Because each w. φ(x) (see below)
needs 75 million operations. What
can be done?
k
y k k
K k s.t. α k > 0K
K K
5 Then When classify you see this with: many callout bubbles on
where
=
K∑=
α
arg k
y k
( xmax
k
⋅ x + 1)
α
a slide it’s time to wrap the author in a
k s.t. α k > 0
k
blanket, gently take him away and murmur
k
f(x,w,b) “someone’s = been sign(w. at the PowerPoint φ(x) -b) for too
long.”
Copyright © 2001, 2003, Andrew W. Moore Support Vector Machines: Slide 58
Only Sm operations (S=#support vectors)
( x )
29