Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.286 20 — An Example Inference Task: ClusteringInitialization. Set K means {m (k) } to random values.Algorithm 20.2. The K-meansclustering algorithm.Assignment step. Each data point n is assigned to the nearest mean.We denote our guess for the cluster k (n) that the point x (n) belongsto by ˆk (n) .ˆk (n) = argmin{d(m (k) , x (n) )}. (20.3)kAn alternative, equivalent representation of this assignment ofpoints to clusters is given by ‘responsibilities’, which are indicatorvariables r (n)k. In the assignment step, we set r(n)kto one if mean kis the closest mean to datapoint x (n) ; otherwise r (n)kis zero.{r (n) 1 ifk=ˆk(n) = k0 if ˆk(n) ≠ k.(20.4)What about ties? – We don’t expect two means to be exactly thesame distance from a data point, but if a tie does happen, ˆk (n) isset to the smallest of the winning {k}.Update step. The model parameters, the means, are adjusted to matchthe sample means of the data points that they are responsible for.m (k) =∑nr (n)kx(n)where R (k) is the total responsibility of mean k,R (k) (20.5)R (k) = ∑ nr (n)k . (20.6)What about means with no responsibilities? – If R (k) = 0, then weleave the mean m (k) where it is.Repeat the assignment step and update step until the assignmentsdo not change.