A Tutorial on Support Vector Machines for Pattern Recognition

13Hencekwk 2 ==n+1Xij=1n+1Xi=1 i j y i y j x i x j = T H i1 ; y ipn +1=n+1Xi=1 i = nR 2 2!p1 ;n +1(37)Note that this is one of those cases where the Lagrange multiplier can remain undetermined(although determining it is trivial). We have now solved the problem, since all the i are clearly positive orzero (in fact the i will only be zero if all training points havethe same class). Note that kwk depends only on the number of positive (negative) polaritypoints, and not on how the class labels are assigned to the points in Eq. (22). This is clearlynot true of w itself, which isgiven byw =nR 2 (n +1)n+1Xi=1y i ;pn +1The margin, M =2=kwk, isthus given byM =x i (38)2Rpn (1 ; (p=(n +1))2 ) : (39)Thus when the number of points n +1iseven, the minimum margin occurs when p =0(equal numbers of positive and negative examples), in which case the margin is M min =2R= p n. If n + 1 is odd, the minimum margin occurs when p = 1, in which case M min =2R(n +1)=(n p n + 2). In both cases, the maximum margin is given by M max = R(n +1)=n.Thus, for example, for the two dimensional simplex consisting of three points lying on S 1(and spanning R 2 ), and with labeling such that not all three points have the same polarity,the maximum and minimum margin are both 3R=2 (see Figure (12)).Note that the results of this Section amount to an alternative, constructive proof that theVC dimension of oriented separating hyperplanes in R n is at least n +1.3.4. Test PhaseOnce we have trained a Support Vector Machine, how canwe use it? We simply determineon which side of the decision boundary (that hyperplane lying half way between H 1 and H 2and parallel to them) a given test pattern x lies and assign the corresponding class label,i.e. we take the class of x to be sgn(w x + b).3.5. The Non-Separable CaseThe above algorithm for separable data, when applied to non-separable data, will nd nofeasible solution: this will be evidenced by the objective function (i.e. the dual Lagrangian)growing arbitrarily large. So how canwe extend these ideas to handle non-separable data?We would like to relax the constraints (10) and (11), but only when necessary, that is, wewould like tointroduce a further cost (i.e. an increase in the primal objective function) fordoing so. This can be done by introducing positive slack variables i i =1 l in theconstraints (Cortes and Vapnik, 1995), which then become: