A Tutorial on Support Vector Machines for Pattern Recognition

2and Smola, 1996). In most of these cases, SVM generalization performance (i.e. error rateson test sets) either matches or is signicantly better than that of competing methods. Theuse of SVMs for density estimation (Weston et al., 1997) and ANOVA decomposition (Stitsonet al., 1997) has also been studied. Regarding extensions, the basic SVMs contain noprior knowledge of the problem (for example, a large class of SVMs for the image recognitionproblem would give the same results if the pixels were rst permuted randomly (witheach image suering the same permutation), an act of vandalism that would leave the bestperforming neural networks severely handicapped) and much work has been done on incorporatingprior knowledge into SVMs (Scholkopf, Burges and Vapnik, 1996 Scholkopf etal., 1998a Burges, 1998). Although SVMs have good generalization performance, they canbe abysmally slow in test phase, a problem addressed in (Burges, 1996 Osuna and Girosi,1998). Recent work has generalized the basic ideas (Smola, Scholkopf and Muller, 1998aSmola and Scholkopf, 1998), shown connections to regularization theory (Smola, Scholkopfand Muller, 1998b Girosi, 1998 Wahba, 1998), and shown how SVM ideas can be incorporatedin a wide range of other algorithms (Scholkopf, Smola and Muller, 1998b Scholkopfet al, 1998c). The reader may also nd the thesis of (Scholkopf, 1997) helpful.The problem which drove the initial development of SVMs occurs in several guises - thebias variance tradeo (Geman and Bienenstock, 1992), capacitycontrol (Guyon et al., 1992),overtting (Montgomery and Peck, 1992) - but the basic idea is the same. Roughly speaking,for a given learning task, with a given nite amount of training data, the best generalizationperformance will be achieved if the right balance is struck between the accuracy attainedon that particular training set, and the \capacity" of the machine, that is, the ability of themachine to learn any training set without error. A machine with too much capacity is likea botanist with a photographic memory who, when presented with a new tree, concludesthat it is not a tree because it has a dierent number of leaves from anything she has seenbefore amachine with too little capacity islike the botanist's lazy brother, who declaresthat if it's green, it's a tree. Neither can generalize well. The exploration and formalizationof these concepts has resulted in one of the shining peaks of the theory of statistical learning(Vapnik, 1979).In the following, bold typeface will indicate vector or matrix quantities normal typefacewill be used for vector and matrix components and for scalars. We will label componentsof vectors and matrices with Greek indices, and label vectors and matrices themselves withRoman indices. Familiarity with the use of Lagrange multipliers to solve problems withequality or inequality constraints is assumed 2 .2. A Bound on the Generalization Performance of a Pattern Recognition LearningMachineThere is a remarkable family of bounds governing the relation between the capacity of alearning machine and its performance 3 . The theory grew out of considerations of under whatcircumstances, and how quickly, the mean of some empirical quantity converges uniformly,as the number of data points increases, to the true mean (that which would be calculatedfrom an innite amount ofdata)(Vapnik, 1979). Let us start with one of these bounds.The notation here will largely follow that of (Vapnik, 1995). Suppose we are given lobservations. Each observation consists of a pair: avector x i 2 R n i =1:::l and theassociated \truth" y i ,given to us by a trusted source. In the tree recognition problem, x imight beavector of pixel values (e.g. n = 256 for a 16x16 image), and y i would be 1 if theimage contains a tree, and -1 otherwise (we use -1 here rather than 0 to simplify subsequentformulae). Now it is assumed that there exists some unknown probability distribution