A Tutorial on Support Vector Machines for Pattern Recognition

39from the set can be found which assigns those labels to the points. The VC dimension of thatset of functions is then dened as the maximum number of points that can be so shattered.However, consider a slightly dierent denition. Let a set of points be shattered by a setof functions if for any choice of labels for the points, a function from the set can be foundwhich assigns the incorrect labels to all the points. Again let the VC dimension of that setof functions be dened as the maximum number of points that can be so shattered.It is in fact this second denition (which we adopt from here on) that enters the VCbound proofs (Vapnik, 1979 Devroye, Gyor and Lugosi, 1996). Of course for functionswhose range is f1g (i.e. all data will be assigned either positive or negative class), the twodenitions are the same. However,ifallpoints falling in some region are simply deemed tobe \errors", or \correct", the two denitions are dierent. As a concrete example, supposewe dene \gap intolerant classiers", which are like gap tolerant classiers, but which labelall points lying in the margin or outside the sphere as errors. Consider again the situation inFigure 12, but assign positive class to all three points. Then a gap intolerant classier withmargin width greater than the ball diameter cannot shatter the points if we use the rstdenition of \shatter", but can shatter the points if we use the second (correct) denition.With this caveat in mind, we now outline how theVC bounds can apply to functions withrange f1 0g, where the label 0 means that the point islabeled \correct." (The boundswill also apply to functions where 0 is dened to mean \error", but the corresponding VCdimension will be higher, weakening the bound, and in our case, making it useless). We willfollow the notation of (Devroye, Gyor and Lugosi, 1996).Consider points x 2 R d , and let p(x) denote a density onR d . Let be a function on R dwith range f1 0g, and let be a set of such functions. Let each x have an associatedlabel y x 2f1g. Let fx 1 x n g be any nite number of points in R d : then we require to have the property that there exists at least one 2 such that (x i ) 2f1g 8x i . Forgiven , dene the set of points A byA = fx : y x =1(x) =;1g[fx : y x = ;1(x) =1g(A.9)We require that the be such that all sets A are measurable. Let A denote the set of allA.Denition: Let x i i = 1 n be n points. We dene the empirical risk for the setfx i g to be n (fx i g) =(1=n)nXi=1Ix i 2A:(A.10)where I is the indicator function. Note that the empirical risk is zero if (x i )=08 x i .Denition: We dene the actual risk for the function to be() =P (x 2 A):(A.11)Note also that those points x for which (x) = 0 do not contribute to the actual risk.Denition: For xed (x 1 x n ) 2 R d ,letN A be the number of dierent setsinffx 1 x n g\A : A 2Ag(A.12)