A Tutorial on Support Vector Machines for Pattern Recognition

17K(x i x j )=e ;kxi;xjk2 =2 2 : (60)In this particular example, H is innite dimensional, so it would not be very easy to workwith explicitly. However, if one replaces x i x j by K(x i x j )everywhere in the trainingalgorithm, the algorithm will happily produce a support vector machine which lives in aninnite dimensional space, and furthermore do so in roughly the same amount oftime itwould take to train on the un-mapped data. All the considerations of the previous sectionshold, since we are still doing a linear separation, but in a dierent space.But how can we use this machine? After all, we need w, and that will live inH also (seeEq. (46)). But in test phase an SVM is used by computing dot products of a given testpoint x with w, or more specically by computing the sign off(x) =XN Si=1 i y i (s i ) (x)+b =XN Si=1 i y i K(s i x)+b (61)where the s i are the support vectors. So again we canavoid computing (x) explicitlyand use the K(s i x) =(s i ) (x) instead.Let us call the space in which the data live, L. (Here and below weuseL as a mnemonicfor \low dimensional", and H for \high dimensional": it is usually the case that the range ofisofmuch higher dimension than its domain). Note that, in addition to the fact that wlives in H, there will in general be no vector in L which maps, via the map , to w. If therewere, f(x) in Eq. (61) could be computed in one step, avoiding the sum (and making thecorresponding SVM N S times faster, where N S is the number of support vectors). Despitethis, ideas along these lines can be used to signicantly speed up the test phase of SVMs(Burges, 1996). Note also that it is easy to nd kernels (for example, kernels which arefunctions of the dot products of the x i in L) such that the training algorithm and solutionfound are independent of the dimension of both L and H.In the next Section we will discuss which functions K are allowable and which arenot.Let us end this Section with a very simple example of an allowed kernel, for which we canconstruct the mapping .Suppose that your data are vectors in R 2 , and you choose K(x i x j )=(x i x j ) 2 . Thenit's easy to nd a space H, and mapping from R 2 to H, suchthat(x y) 2 =(x) (y):we choose H = R 3 and(x) =0@ x2 1 p2 x1 x 2x 2 21A (62)(note that here the subscripts refer to vector components). For data in L dened on thesquare [;1 1] [;1 1] 2 R 2 (a typical situation, for grey level image data), the (entire)image of is shown in Figure 8. This Figure also illustrates how to think of this mapping:the image of may live in a space of very high dimension, but it is just a (possibly verycontorted) surface whose intrinsic dimension 12 is just that of L.Note that neither the mapping nor the space H are unique for a given kernel. We couldequally well have chosen H to again be R 3 and0 1(x) = p 1 @ (x2 1 ; x 2 2)2x 1 x 2A (63)2 (x 2 + 1 x2) 2