Student Notes To Accompany MS4214: STATISTICAL INFERENCE

2.8 Optimality Properties of the MLE
Suppose that an experiment consists of measuring random variables x1, x2, . . . , xn which
are iid with probability distribution depending on a parameter θ. Let ˆ θ be the MLE
of θ. Define
W1 = � E[I(θ)]( ˆ θ − θ)
W2 = � I(θ)( ˆ θ − θ)
�
W3 = E[I( ˆ θ)]( ˆ θ − θ)
�
W4 = I( ˆ θ)( ˆ θ − θ).
Then, W1, W2, W3, and W4 are all random variables and, as n → ∞, the probabilistic
behaviour of each of W1, W2, W3, and W4 is well approximated by that of a N(0, 1)
random variable. Then, since E[W1] ≈ 0, we have that E[ ˆ θ] ≈ θ and so ˆ θ is approx-
imately unbiased. Also Var[W1] ≈ 1 implies that Var[ ˆ θ] ≈ (E[I(θ)]) −1 and so ˆ θ is
approximately efficient.
Let the data X have probability distribution g(X; θ) where θ = (θ1, θ2, . . . , θm) is a
vector of m unknown parameters. Let I(θ) be the m×m information matrix as defined
above and let E[I(θ)] be the m × m matrix obtained by replacing the elements of I(θ)
by their expected values. Let ˆ θ be the MLE of θ. Let CRLBr be the rth diagonal
element of [E[I(θ)]] −1 . For r = 1, 2, . . . , m, define W1r = ( ˆ θr − θr)/ √ CRLBr. Then, as
n → ∞, W1r behaves like a standard normal random variable.
Suppose we define W2r by replacing CRLBr by the rth diagonal element of the
matrix [I(θ)] −1 , W3r by replacing CRLBr by the rth diagonal element of the matrix
[EI( ˆ θ)] −1 and W4r by replacing CRLBr by the rth diagonal element of the matrix
[I( ˆ θ)] −1 . Then it can be shown that as n → ∞, W2r, W3r, and W4r all behave like
standard normal random variables.
2.9 Data Reduction
Definition 2.11 (Sufficiency). Consider a statistic T = t(X) that summarises the
data so that no information about θ is lost. Then we call t(X) a sufficient statistic. �
Example 2.12. T = t(X) = ¯ X is sufficient for µ when Xi ∼ iid N(µ, σ 2 ). �
To better understand the motivation behind the concept of sufficiency consider
three independent Binomial trials where θ = P (X = 1).
39