A Tutorial on Support Vector Machines for Pattern Recognition

33with solution given byC = X i i (x i ): (92)Thus the problem is very similar to that of support vector training, and in fact the codefor the latter is easily modied to solve the above problem. Note that we were in a sense\lucky", because the above analysis shows us that there exists an expansion (92) for thecenter there is no a priori reason why we should expect that the center of the sphere in Hshould be expressible in terms of the mapped training data in this way. The same can besaid of the solution for the support vector problem, Eq. (46). (Had we chosen some othergeometrical construction, we might nothave been so fortunate. Consider the smallest areaequilateral triangle containing two given points in R 2 . If the points' position vectors arelinearly dependent, the center of the triangle cannot be expressed in terms of them.)7.4. A Bound from Leave-One-Out(Vapnik, 1995) gives an alternative bound on the actual risk of support vector machines:E[P (error)] E[Number of support vectors]Number of training samples (93)where P (error) is the actual risk for a machine trained on l ; 1 examples, E[P (error)]is the expectation of the actual risk over all choices of training set of size l ; 1, andE[Number of support vectors] is the expectation of the number of support vectors over allchoices of training sets of size l. It's easy to see how this bound arises: consider the typicalsituation after training on a given training set, shown in Figure 13.Figure 13.Support vectors (circles) can become errors (cross) after removal and re-training (the dotted linedenotes the new decision surface).We can get an estimate of the test error by removing one of the training points, re-training,and then testing on the removed point and then repeating this, for all training points. Fromthe support vector solution we know that removing any training points that are not supportvectors (the latter include the errors) will have no eect on the hyperplane found. Thusthe worst that can happen is that every support vector will become an error. Taking theexpectation over all such training sets therefore gives an upper bound on the actual risk,for training sets of size l ; 1.