Foundations of Data Science

y
x
φ
Figure 6.4: Data that is not linearly separable in the input space R 2 but that is linearly
separable in the “φ-space,” φ(x) = (1, √ 2x 1 , √ 2x 2 , x 2 1, √ 2x 1 x 2 , x 2 2), corresponding to the
kernel function K(x t y) = (1 + x 1 x 2 + y 1 y 2 ) 2 .
d = 2, k = 2 we have (using x i to denote the ith coordinate of x):
K(x, x ′ ) = (1 + x 1 x ′ 1 + x 2 x ′ 2) 2
= 1 + 2x 1 x ′ 1 + 2x 2 x ′ 2 + x 2 1x ′2
1 + 2x 1 x 2 x ′ 1x ′ 2 + x 2 2x ′2
2
= φ(x) T φ(x ′ )
for φ(x) = (1, √ 2x 1 , √ 2x 2 , x 2 1, √ 2x 1 x 2 , x 2 2). Notice also that a linear separator in this
space could correspond to a more complicated decision boundary such as an ellipse in
the original space. For instance, the hyperplane φ(x) T w ∗ = 0 for w ∗ = (−4, 0, 0, 1, 0, 1)
corresponds to the circle x 2 1 + x 2 2 = 4 in the original space, such as in Figure 6.4.
The point of this is that if in the higher-dimensional “φ-space” there is a w ∗ such that
the bound of Theorem 6.9 is small, then the algorithm will perform well and make few
mistakes. But the nice thing is we didn’t have to computationally perform the mapping φ!
So, how can we view the Perceptron algorithm as only interacting with data via dotproducts?
Notice that w is always a linear combination of data points. For example, if we
made mistakes on the first, second and fifth examples, and these examples were positive,
positive, and negative respectively, we would have w = x 1 +x 2 −x 5 . So, if we keep track of
w this way, then to predict on a new example x t , we can write x T t w = x T t x 1 +x T t x 2 −x T t x 5 .
So if we just replace each of these dot-products with “K”, we are running the algorithm
as if we had explicitly performed the φ mapping. This is called “kernelizing” the algorithm.
Many different pairwise functions on examples are legal kernel functions. One easy
way to create a kernel function is by combining other kernel functions together, via the
following theorem.
Theorem 6.10 Suppose K 1 and K 2 are kernel functions. Then
203