Foundations of Data Science

1. For any constant c ≥ 0, cK 1 is a legal kernel. In fact, for any scalar function f,
the function K 3 (x, x ′ ) = f(x)f(x ′ )K 1 (x, x ′ ) is a legal kernel.
2. The sum K 1 + K 2 , is a legal kernel.
3. The product, K 1 K 2 , is a legal kernel.
You will prove Theorem 6.10 in Exercise 6.9. Notice that this immediately implies that
the function K(x, x ′ ) = (1+x T x ′ ) k is a legal kernel by using the fact that K 1 (x, x ′ ) = 1 is
a legal kernel, K 2 (x, x ′ ) = x T x ′ is a legal kernel, then adding them, and then multiplying
that by itself k times. Another popular kernel is the Gaussian kernel, defined as:
K(x, x ′ ) = e −c|x−x′ | 2 .
If we think of a kernel as a measure of similarity, then this kernel defines the similarity
between two data objects as a quantity that decreases exponentially with the squared
distance between them. The Gaussian kernel can be shown to be a true kernel function
by first writing it as f(x)f(x ′ )e 2cxT x ′ for f(x) = e −c|x|2 and then taking the Taylor
expansion of e 2cxT x ′ , applying the rules in Theorem 6.10. Technically, this last step requires
considering countably infinitely many applications of the rules and allowing for
infinite-dimensional vector spaces.
6.7 Online to Batch Conversion
Suppose we have an online algorithm with a good mistake bound, such as the Perceptron
algorithm. Can we use it to get a guarantee in the distributional (batch) learning setting?
Intuitively, the answer should be yes since the online setting is only harder. Indeed, this
intuition is correct. We present here two natural approaches for such online to batch
conversion.
Conversion procedure 1: Random Stopping. Suppose we have an online algorithm
A with mistake-bound M. Say we run the algorithm in a single pass on a sample S of size
M/ɛ. Let X t be the indicator random variable for the event that A makes a mistake on the
tth example. Since ∑ |S|
t=1 X t ≤ M for any set S, we certainly have that E[ ∑ |S|
t=1 X t] ≤ M
where the expectation is taken over the random draw of S from D |S| . By linearity of
expectation, and dividing both sides by |S| we therefore have:
1
|S|
|S|
∑
t=1
E[X t ] ≤ M/|S| = ɛ. (6.1)
Let h t denote the hypothesis used by algorithm A to predict on the tth example. Since
the tth example was randomly drawn from D, we have E[err D (h t )] = E[X t ]. This means
that if we choose t at random from 1 to |S|, i.e., stop the algorithm at a random time, the
expected error of the resulting prediction rule, taken over the randomness in the draw of
S and the choice of t, is at most ɛ as given by equation (6.1). Thus we have:
204