Student Notes To Accompany MS4214: STATISTICAL INFERENCE

Problem 2.4. Consider X1, . . . , Xn where Xi ∼ N(θ, σ 2 ) and σ is known. Three
estimators of θ are ˆ θ1 = ¯ X = 1
n
� n
i=1 Xi, ˆ θ2 = X1, and ˆ θ3 = (X1 + ¯ X)/2. Pick one.
Solution. E( ˆ θ1) = 1
n [E(X1)+· · ·+E(Xn)] = 1
1
[θ+· · ·+θ] = [nθ] = θ, (unbiased). Next
n n
E( ˆ θ2) = E(X1) = θ, (unbiased). Finally E( ˆ θ3) = 1
2E � n+1
n X1 + 1
n [X2 + · · · + Xn] � =
�
1 n+1
2 n E(X1) + 1
n [E(X2) + · · · + E(Xn)] � = 1 n+1 n−1
{ θ + θ} = θ, (unbiased). All three
2 n n
estimators are unbiased. Although desirable from a frequentist standpoint, unbiased-
ness is not a property that helps us choose between estimators. To do this we must
examine some measure of loss like the mean squared error. For class of estimators
that are unbiased, the mean squared error will be equal to the estimation variance.
Calculate Var( ˆ θ1) = 1
n 2 [Var(X1) + · · · + Var(Xn)] = 1
n 2 [σ 2 + · · · + σ 2 ] = 1
n 2 [nσ 2 ] = 1
n σ2 .
Trivially Var( ˆ θ2) = Var(X1) = σ 2 . Finally Var( ˆ θ3) = (σ 2 /n + σ 2 )/4 + 2Cov( ¯ X, X1).
So ¯ X appears “best” in the sense that Var( ˆ θ) is smallest among these three unbiased
estimators.
Problem 2.5. Consider X1, . . . , Xn to be independent random variables with means
E(Xi) = µ + βi and variances Var(Xi) = σ 2 i . Such a situation could arise when Xi are
estimators of µ obtained from independent sources and βi is the bias of the estimator
Xi. Consider pooling the estimators of µ into a common estimator using the linear
combination ˆµ = w1X1 + w2X2 + · · · + wnXn.
(i) If the estimators are unbiased, show that ˆµ is unbiased if and only if � wi = 1.
(ii) In the case when the estimators are unbiased, show that ˆµ has minimum variance
when the weights are inversely proportional to the variances σ 2 i .
(iii) Show that the variance of ˆµ for optimal weights wi is Var(ˆµ) = 1/ �
i σ−2
i .
(iv) Consider the case when estimators may be biased. Find the mean square error
of the optimal linear combination obtained above, and compare its behaviour as
n → ∞ in the biased and unbiased case, when σ 2 i = σ 2 , i = 1, . . . , n.
Solution. E(ˆµ) = E(w1X1 + · · · + wnXn) = �
i wiE(Xi) = �
i wiµ = µ �
i wi so ˆµ is
unbiased if and only if �
i wi = 1. The variance of our estimator is Var(ˆµ) = �
i w2 i σ 2 i ,
which should be minimized subject to the constraint �
i wi = 1. Differentiating the
Lagrangian L = �
i w2 i σ 2 i − λ ( �
i wi − 1) with respect to wi and setting equal to zero
yields 2wiσ 2 i = λ ⇒ wi ∝ σ −2
i so that wi = σ −2
i /(�
j σ−2 j
). Then, for optimal weights we
get Var(ˆµ) = �
i w2 i σ2 i = ( �
i σ−4 i σ2 i )/( �
i σ−2 i )2 = 1/( �
i σ−2 i ). When σ2 i = σ2 we have
that Var(ˆµ) = σ2 /n which tends to zero for n → ∞ whereas bias(ˆµ) = � βi/n = ¯ β is
equal to the average bias and MSE(ˆµ) = σ 2 + ¯ β 2 . Therefore the bias tends to dominate
the variance as n gets larger, which is very unfortunate.
13