Student Notes To Accompany MS4214: STATISTICAL INFERENCE

Given any estimator ˆ θ we can calculate its expected value for each possible value of
θ ∈ Θ. An estimator is said to be unbiased if this expected value is identically equal to
θ. If an estimator is unbiased then we can conclude that if we repeat the experiment
an infinite number of times with θ fixed and calculate the value of the estimator each
time then the average of the estimator values will be exactly equal to θ. From the
frequentist viewpoint this is a desireable property and so, where possible, frequentists
use unbiased estimators.
Definition 2.1 (The Frequentist philosophy). To evaluate the usefulness of an
estimator ˆ θ = ˆ θ(x) of θ, examine the properties of the random variable ˆ θ = ˆ θ(X). �
Definition 2.2 (Unbiased estimators). An estimator ˆ θ = ˆ θ(X) is said to be unbi-
ased for a parameter θ if it equals θ in expectation:
E[ ˆ θ(X)] = E( ˆ θ) = θ.
Intuitively, an unbiased estimator is ‘right on target’. �
Definition 2.3 (Bias of an estimator). The bias of an estimator ˆ θ = ˆ θ(X) of θ is
defined as bias( ˆ θ) = E[ ˆ θ(X) − θ]. �
Definition 2.4 (Bias corrected estimators). If bias( ˆ θ) is of the form cθ, then
(obviously) ˜ θ = ˆ θ/(1 + c) is unbiased for θ. Likewise, if bias( ˆ θ) = θ + c, then ˜ θ = ˆ θ − c
is unbiased for θ. In such situations we say that ˜ θ is a biased corrected version of ˆ θ. �
Definition 2.5 (Unbiased functions). More generally ˆg(X) is said to be unbiased
for a function g(θ) if E[ˆg(X)] = g(θ). �
Note that even if ˆ θ is an unbiased estimator of θ, g( ˆ θ) will generally not be an
unbiased estimator of g(θ) unless g is linear or affine. This limits the importance of the
notion of unbiasedness. It might be at least as important that an estimator is accurate
in the sense that its distribution is highly concentrated around θ.
Is unbiasedness a good thing? Unbiasedness is important when combining esti-
mates, as averages of unbiased estimators are unbiased (see the review exercises at the
end of this chapter). For example, when combining standard deviations s1, s2, . . . , sk
with degrees of freedom df1, . . . , dfk we always average their squares
�
df1s
¯s =
2 1 + · · · + dfks2 k
df1 + · · · + dfk
as s 2 i are unbiased estimators of the variance σ 2 , whereas si are not unbiased estimators
of σ (see the review exercises). Be careful when averaging biased estimators! It may
well be appropriate to make a bias-correction before averaging.
11