A Tutorial on Support Vector Machines for Pattern Recognition

358. LimitationsPerhaps the biggest limitation of the support vector approach lies in choice of the kernel.Once the kernel is xed, SVM classiers have only one user-chosen parameter (the errorpenalty), but the kernel is a very big rug under which tosweep parameters. Some work hasbeen done on limiting kernels using prior knowledge (Scholkopf et al., 1998a Burges, 1998),but the best choice of kernel for a given problem is still a research issue.A second limitation is speed and size, both in training and testing. While the speedproblem in test phase is largely solved in (Burges, 1996), this still requires two trainingpasses. Training for very large datasets (millions of support vectors) is an unsolved problem.Discrete data presents another problem, although with suitable rescaling excellent resultshave nevertheless been obtained (Joachims, 1997). Finally, although some work has beendone on training a multiclass SVM in one step 24 , the optimal design for multiclass SVMclassiers is a further area for research.9. ExtensionsWe very briey describe two of the simplest, and most eective, methods for improving theperformance of SVMs.The virtual support vector method (Scholkopf, Burges and Vapnik, 1996 Burges andScholkopf, 1997), attempts to incorporate known invariances of the problem (for example,translation invariance for the image recognition problem) by rst training a system, andthen creating new data by distorting the resulting support vectors (translating them, in thecase mentioned), and nally training a new system on the distorted (and the undistorted)data. The idea is easy to implement and seems to work better than other methods forincorporating invariances proposed so far.The reduced set method (Burges, 1996 Burges and Scholkopf, 1997) was introduced toaddress the speed of support vector machines in test phase, and also starts with a trainedSVM. The idea is to replace the sum in Eq. (46) by a similar sum, where instead of supportvectors, computed vectors (which are not elements of the training set) are used, and insteadof the i , a dierent set of weights are computed. The number of parameters is chosenbeforehand to give the speedup desired. The resulting vector is still a vector in H, andthe parameters are found by minimizing the Euclidean norm of the dierence between theoriginal vector w and the approximation to it. The same technique could be used for SVMregression to nd much more ecient function representations (which could be used, forexample, in data compression).Combining these two methods gave a factor of 50 speedup (while the error rate increasedfrom 1.0% to 1.1%) on the NIST digits (Burges and Scholkopf, 1997).10. ConclusionsSVMs provide a new approach to the problem of pattern recognition (together with regressionestimation and linear operator inversion) with clear connections to the underlyingstatistical learning theory. They dier radically from comparable approaches such as neuralnetworks: SVM training always nds a global minimum, and their simple geometric interpretationprovides fertile ground for further investigation. An SVM is largely characterizedby thechoice of its kernel, and SVMs thus link the problems they are designed for with alarge body of existing work on kernel based methods. I hope that this tutorial will encouragesome to explore SVMs for themselves.