Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.20.2: Soft K-means clustering 289has an equal vote with all the other points in that cluster, and no vote in anyother clusters.20.2 Soft K-means clusteringThese criticisms of K-means motivate the ‘soft K-means algorithm’, algorithm20.7. The algorithm has one parameter, β, which we could term thestiffness.point x (n) is givenmeans. We callthe responsibility −β d(m (k)∑k ′ exp ( −β d(m (k′)responsibilities for theparameters, the means,data points that∑r (n)kx(n)m (k) n=R (k)responsibility of meanR (k) = ∑ r (n)kn, of assignment’which x (n)responsibility of(20.7)matchresponsible for.(20.8)Algorithm 20.7. Soft K-meansalgorithm, version 1.Assignment step. Each data a soft ‘degreeto each of the the degree to is assigned to cluster k r (n)k(thecluster k for point n).r (n)x (n) ) )k= )., x (n) )The sum of the K nth point is 1.Update step. The model are adjusted tothe sample means of the they arewhere R (k) is the total k,Notice the similarity of this soft K-means algorithm to the hard K-meansalgorithm 20.2. The update step is identical; the only difference is that theresponsibilities r (n)kcan take on values between 0 and 1. Whereas the assignmentˆk (n) in the K-means algorithm involved a ‘min’ over the distances, therule for assigning the responsibilities is a ‘soft-min’ (20.7).⊲ Exercise 20.2. [2 ] Show that as the stiffness β goes to ∞, the soft K-means algorithmbecomes identical to the original hard K-means algorithm, exceptfor the way in which means with no assigned points behave. Describewhat those means do instead of sitting still.Dimensionally, the stiffness β is an inverse-length-squared, so we can associatea lengthscale, σ ≡ 1/ √ β, with it. The soft K-means algorithm isdemonstrated in figure 20.8. The lengthscale is shown by the radius of thecircles surrounding the four means. Each panel shows the final fixed pointreached for a different value of the lengthscale σ.20.3 ConclusionAt this point, we may have fixed some of the problems with the original K-means algorithm by introducing an extra complexity-control parameter β. Buthow should we set β? And what about the problem of the elongated clusters,