Theory of Statistics - George Mason University

474 6 Statistical Inference Based on Likelihood
qk(x, θ) + c(x, θ (k−1) ) ≤ lX(θ; x),
where c(x, θ (k−1) ) is constant with respect to θ.
Therefore for given θ (k−1) and any x,
is a minorizing function for lX(θ; x).
g(θ) = lX(θ (k−1) ; X) − qk(x, θ (k−1) )
Alternative Ways of Performing the Computations
There are two kinds of computations that must be performed in each iteration:
• E step : compute qk(x, θ) = EU|x,θ (k−1) lc(θ; x, U) .
• M step : determine θ (k) to maximize qk(x, θ), subject to any constraints
on acceptable values of θ.
There are obviously various ways to perform each of these computations.
A number of papers since 1977 have suggested specific methods for the
computations.
For each specification of a method for doing the computations or each little
modification, a new name is given, just as if it were a new idea:
GEM, ECM, ECME, AECM, GAECM, PXEM, MCEM, AEM, EM1, SEM
E Step
There are various ways the expectation step can be carried out.
In the happy case of a “nice” distribution, the expectation can be computed
in closed form. Otherwise, computing the expectation is a numerical
quadrature problem. There are various procedures for quadrature, including
Monte Carlo.
Some people have called an EM method that uses Monte Carlo to evaluate
the expectation an MCEM method. (If a Newton-Cotes method is used,
however, we do not call it an NCEM method!) The additional Monte Carlo
computations add a lot to the overall time required for convergence of the EM
method.
An additional problem in using Monte Carlo in the expectation step may
be that the distribution of C is difficult to simulate. The convergence criterion
for optimization methods that involve Monte Carlo generally should be tighter
than for deterministic methods.
M Step
For the maximization step, there are even more choices.
Theory of Statistics c○2000–2013 James E. Gentle