Theory of Statistics - George Mason University

460 6 Statistical Inference Based on Likelihood
The derivatives of the log-likelihood function relate directly to useful concepts
in statistical inference. If it exists, the derivative of the log-likelihood
is the relative rate of change, with respect to the parameter placeholder θ, of
the probability density function at a fixed observation. If θ is a scalar, some
positive function of the derivative, such as its square or its absolute value, is
obviously a measure of the effect of change in the parameter, or of change
in the estimate of the parameter. More generally, an outer product of the
derivative with itself is a useful measure of the changes in the components of
the parameter:
(k)
∇lL θ ; x (k)
∇lL θ ; x T .
Notice that the average of this quantity with respect to the probability density
of the random variable X,
(k)
I(θ1 ; X) = Eθ1 ∇lL θ ; X
(k)
∇lL θ ; X T
, (6.30)
is the information matrix for an observation on Y about the parameter θ.
If θ is a scalar, the square of the first derivative is the negative of the
second derivative,
or, in general,
∂
∂θ lL(θ
2 ; x)
(k)
∇lL θ ; x
(k)
∇lL θ ; x T MLEs in Exponential Families
= − ∂2
∂θ 2 lL(θ ; x),
= −HlL
θ (k) ; x . (6.31)
If X has a distribution in the exponential class and we write its density in the
natural or canonical form, the likelihood has the form
L(η ; x) = exp(η T T(x) − ζ(η))h(x). (6.32)
The log-likelihood equation is particularly simple:
T(x) − ∂ζ(η)
∂η
Newton’s method for solving the likelihood equation is
= 0. (6.33)
η (k) = η (k−1) 2 ∂ ζ(η)
−
∂η(∂η) T
−1
η=η (k−1) T(x) − ∂ζ(η)
η=η (k−1)
∂η
Theory of Statistics c○2000–2013 James E. Gentle