Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.488 40 — Capacity of a Single Neuronof the binary functions on 4 points in 3 dimensions cannot be realized by alinear threshold function.We have now filled in the values of T (N, K) shown in table 40.6. Can weobtain any insights into our derivation of T (4, 3) in order to fill in the rest ofthe table for T (N, K)? Why was T (4, 3) greater than T (3, 3) by six?Six is the number of regions that the new hyperplane bisected in w-space(figure 40.7a b). Equivalently, if we look in the K −1 dimensional subspacethat is the Nth hyperplane, that subspace is divided into six regions by theN −1 previous hyperplanes (figure 40.7c). Now this is a concept we have metbefore. Compare figure 40.7c with figure 40.4b. How many regions are createdby N − 1 hyperplanes in a K−1 dimensional space? Why, T (N −1, K−1), ofcourse! In the present case N = 4, K = 3, we can look up T (3, 2) = 6 in theprevious section. SoT (4, 3) = T (3, 3) + T (3, 2). (40.4)Recurrence relation for any N, KGeneralizing this picture, we see that when we add an Nth hyperplane in Kdimensions, it will bisect T (N −1, K−1) of the T (N −1, K) regions that werecreated by the previous N −1 hyperplanes. Therefore, the total number ofregions obtained after adding the Nth hyperplane is 2T (N −1, K −1) (sinceT (N−1, K−1) out of T (N−1, K) regions are split in two) plus the remainingT (N −1, K) − T (N −1, K−1) regions not split by the Nth hyperplane, whichgives the following equation for T (N, K):T (N, K) = T (N −1, K) + T (N −1, K −1). (40.5)Now all that remains is to solve this recurrence relation given the boundaryconditions T (N, 1) = 2 and T (1, K) = 2.Does the recurrence relation (40.5) look familiar? Maybe you rememberbuilding Pascal’s triangle by adding together two adjacent numbers in one rowto get the number below. The N, K element of Pascal’s triangle is equal to( N N!C(N, K) ≡ ≡K)(N − K)!K! . (40.6)KN 0 1 2 3 4 5 6 7Table 40.8. Pascal’s triangle.0 11 1 12 1 2 13 1 3 3 14 1 4 6 4 15 1 5 10 10 5 1Combinations ( NK)satisfy the equationC(N, K) = C(N −1, K −1) + C(N −1, K), for all N > 0. (40.7)[Here we are adopting the convention that ( NK)≡ 0 if K > N or K < 0.]So ( NK)satisfies the required recurrence relation (40.5). This doesn’t meanT (N, K) = ( NK), since many functions can satisfy one recurrence relation.