Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.40.3: Counting threshold functions 485w 2(1)x 2x 1x (1) (b)w 1(0)Figure 40.2. One data point in atwo-dimensional input space, andthe two regions of weight spacethat give the two alternativelabellings of that point.(a)40.3 Counting threshold functionsLet us denote by T (N, K) the number of distinct threshold functions on Npoints in general position in K dimensions. We will derive a formula forT (N, K).To start with, let us work out a few cases by hand.In K = 1 dimension, for any NThe N points lie on a line. By changing the sign of the one weight w 1 we canlabel all points on the right side of the origin 1 and the others 0, or vice versa.Thus there are two distinct threshold functions. T (N, 1) = 2.With N = 1 point, for any KIf there is just one point x (1) then we can realize both possible labellings bysetting w = ±x (1) . Thus T (1, K) = 2.In K = 2 dimensionsIn two dimensions with N points, we are free to spin the separating line aroundthe origin. Each time the line passes over a point we obtain a new function.Once we have spun the line through 360 degrees we reproduce the functionwe started from. Because the points are in general position, the separatingplane (line) crosses only one point at a time. In one revolution, every pointis passed over twice. There are therefore 2N distinct threshold functions.T (N, 2) = 2N.Comparing with the total number of binary functions, 2 N , we may notethat for N ≥ 3, not all binary functions can be realized by a linear thresholdfunction. One famous example of an unrealizable function with N = 4 andK = 2 is the exclusive-or function on the points x = (±1, ±1). [These pointsare not in general position, but you may confirm that the function remainsunrealizable even if the points are perturbed into general position.]In K = 2 dimensions, from the point of view of weight spaceThere is another way of visualizing this problem. Instead of visualizing aplane separating points in the two-dimensional input space, we can considerthe two-dimensional weight space, colouring regions in weight space differentcolours if they label the given datapoints differently. We can then count thenumber of threshold functions by counting how many distinguishable regionsthere are in weight space. Consider first the set of weight vectors in weight