Theory of Statistics - George Mason University

214 3 Basic Statistical Theory
closeness, and equivariance under transformations. In this section we consider
these properties only in the context of estimation, but the same properties
have meaning in other types of statistical inference, although the specific definition
may be different. “Good” can be defined either in terms of various
measures associated with an estimator (bias, mean squared error, Pitman
closeness), or as a property that the estimator either possesses or does not
(equivariance).
Bias
The bias of T(X) for estimating g(θ) is
E(T(X)) − g(θ). (3.14)
One of the most commonly required desirable properties of point estimators
is unbiasedness.
Definition 3.1 (unbiased point estimator)
The estimator T(X) is unbiased for g(θ) if
E(T(X)) = g(θ) ∀θ ∈ Θ.
Unbiasedness is a uniform property of the expected value.
We can also define other types of unbiasedness in terms of other aspects
of a probability distribution. For example, an estimator whose median is the
estimand is said to be median-unbiased.
Unbiasedness has different definitions in other settings (estimating functions,
for example, see page 251) and for other types of statistical inference (for
example, testing, see page 292, and determining confidence sets, see page 296),
but the meanings are similar.
If two estimators are unbiased, we would reasonably prefer one with smaller
variance.
Mean Squared Error
Another measure of the goodness of a scalar estimator is the mean squared
error or MSE,
MSE(T(x)) = E((T(X) − g(θ)) 2 ), (3.15)
which is the square of the bias plus the variance:
MSE(T(x)) = (E(T(X)) − g(θ)) 2 + V(T(X)). (3.16)
An example due to C. R. Rao (Rao, 1981) causes us to realize that we
often face a dilemma in finding a good estimate. A “bad” estimator may have
a smaller MSE than a “good” estimator.
Theory of Statistics c○2000–2013 James E. Gentle