A Tutorial on Support Vector Machines for Pattern Recognition

37Pi =1 0 i 1. Then w x+b = P i ifwa i + bg > 0.P may be written x =i ia i TSimilarly, for points y 2 C B , w y + b < 0. Hence C A CB = , since otherwise wewould be able to nd a point x = y which simultaneously satises both inequalities.Theorem 1: Consider some set of m points in R n . Choose any one of the points asorigin. Then the m points can be shattered by oriented hyperplanes if and only if theposition vectors of the remaining points are linearly independent.Proof: Label the origin O, and assume that the m ; 1 position vectors of the remainingpoints are linearly independent. Consider any partition of the m points into two subsets,S 1 and S 2 ,oforderm 1 and m 2 respectively, sothatm 1 + m 2 = m. Let S 1 be the subsetcontaining O. Then the convex hull C 1 of S 1 is that set of points whose position vectors xsatisfyx =Xm 1i=1 i s 1i Xm 1i=1 i =1 i 0 (A.1)where the s 1i are the position vectors of the m 1 points in S 1 (including the null positionvector of the origin). Similarly, the convex hull C 2 of S 2 is that set of points whose positionvectors x satisfyx =Xm 2i=1 i s 2i Xm 2i=1 i =1 i 0 (A.2)where the s 2i are the position vectors of the m 2 points in S 2 . Now suppose that C 1 andC 2 intersect. Then there exists an x 2 R n which simultaneously satises Eq. (A.1) and Eq.(A.2). Subtracting these equations gives a linear combination of the m ; 1 non-null positionvectors which vanishes, which contradicts the assumption of linear independence. By thelemma, since C 1 and C 2 do not intersect, there exists a hyperplane separating S 1 and S 2 .Since this is true for any choice of partition, the m points can be shattered.It remains to show that if the m;1 non-null position vectors are not linearly independent,then the m points cannot be shattered by oriented hyperplanes. If the m;1 position vectorsare not linearly independent, then there exist m ; 1numbers, i ,suchthatm;1Xi=1 i s i =0(A.3)If all the i are of the same sign, then we can scale them so that i 2 [0 1] and P i i =1.Eq. (A.3) then states that the origin lies in the convex hull of the remaining points hence,by the lemma, the origin cannot be separated from the remaining points by ahyperplane,and the points cannot be shattered.If the i are not all of the same sign, place all the terms with negative i on the right:Xj2I 1j j js j = X k2I 2j k js k(A.4)where I 1 , I 2 are the indices of the corresponding partition of SnO (i.e. of the set S with theorigin removed). Now scale this equation so that either P j2I 1j j j =1and P k2I 2j k j1,or P j2I 1j j j 1 and P k2I 2j k j =1. Suppose without loss of generality thatthe latterholds. Then the left hand side of Eq. (A.4) is the position vector of a point lying in the