Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
304 22 — Maximum Likelihood and Clustering
Assignment step. The responsibilities are
r (n)
k =
1
πk
( √
exp −
2πσk) I 1
σ2 d(m
k
(k) , x (n)
)
1
k ′ πk
( √
2πσk ′ ) I exp − 1
σ2 k ′
d(m (k′ (22.22)
) (n)
, x )
where I is the dimensionality of x.
Update step. Each cluster’s parameters, m (k) , πk, and σ2 k , are adjusted
to match the data points that it is responsible for.
σ 2 k =
m (k) =
n
n
r (n)
k x(n)
R (k)
r (n)
k (x(n) − m (k) ) 2
IR (k)
πk = R(k)
k R(k)
where R (k) is the total responsibility of mean k,
R (k) =
r (n)
k =
n
(22.23)
(22.24)
(22.25)
r (n)
k . (22.26)
t = 0 t = 1 t = 2 t = 3 t = 9
t = 0 t = 1 t = 10 t = 20 t = 30 t = 35
πk I i=1
I
1
√ exp − (m
(k)
2πσ i
i=1
(k)
i − x(n)
i ) 2
2(σ (k)
i ) 2
k ′ (numerator, with k′ in place of k)
σ 2(k) i =
n
r (n)
k (x(n)
i − m(k)
i ) 2
R (k)
(22.27)
(22.28)
Algorithm 22.2. The soft K-means
algorithm, version 2.
Figure 22.3. Soft K-means
algorithm, with K = 2, applied
(a) to the 40-point data set of
figure 20.3; (b) to the little ’n’
large data set of figure 20.5.
Algorithm 22.4. The soft K-means
algorithm, version 3, which
corresponds to a model of
axis-aligned Gaussians.