Sufficiency Fisher-Neyman's factorization theorem

Sufficiency
Motivation: Bernoulli distribution
If X 1 , ...., X n are independent Bernoulli-distributed random variables with expected
value p, then the sum T(X) = X 1 + ... + X n is a sufficient statistic for p (here 'success'
corresponds to X i = 1 and 'failure' to X i = 0; so T is the total number of successes)
If we know the total T(X), then the precise x i will not give any further information of
the parameter p.
Sir Ronald Fisher writes, "A statistic satisfies the criterion of sufficiency when no
other statistic which can be calculated from the same sample provides any additional
information as to the value of the parameter to be estimated." [1] This is equivalent to
the more contemporary definition that the distribution of a sample is independent of
the underlying parameter(s) the statistic is sufficient for, conditional on the value of
the sufficient statistic. Both the statistic and the underlying parameters can be vectors.
The concept is most general when defined as follows:
Definition
A statistic T(X) is sufficient for underlying parameter θ precisely if
the conditional probability distribution of the data X, given the statistic T(X), is
independent of the parameter θ.
There must exist a sufficient statistics. T(X) = (X 1 , ...., X n ), T(X) = (X (1) , ...., X (n) ) are
sufficient statistics.
Example
As an example, the sample mean is sufficient for the mean (&mu); of a normal
distribution with known variance. Once the sample mean is known, no further
information about µ can be obtained from the sample itself.
Fisher-Neyman's factorization theorem
Fisher's factorization theorem or factorization criterion
If the likelihood function of X is L θ (x), then T is sufficient for θ if and only if
functions g and h can be found such that
Lθ ( x)
= h(
x)
gθ
( T ( x))
.
i.e. the likelihood L can be factored into a product such that one factor, h, does not
depend on θ and the other factor, which does depend on θ, depends on x only through
T(x).
Interpretation
An implication of the theorem is that when using likelihood-based inference, two sets
of data yielding the same value for the sufficient statistic T(X) will always yield the
same inferences about θ. By the factorization criterion, the likelihood's dependence on

θ is only in conjunction with T(X). As this is the same in both cases, the dependence
on θ will be the same as well, leading to identical inferences.
Example
Binomial distribution p371 Ex2.7
Exponential distribution p371 Ex2.8
Normal distribution p372 Ex2.10
Uniform distribution
If X 1 , ...., X n are independent and uniformly distributed on the interval [0,θ], then T(X)
= max(X 1 , ...., X n ) is sufficient for θ.
If X 1 , ...., X n are independent and uniformly distributed on the interval [a,b], then T(X)
= (min(X 1 , ...., X n ), max(X 1 , ...., X n )) is sufficient for [a,b] p373 Ex2.11.
Poisson distribution
If X 1 , ...., X n are independent and have a Poisson distribution with parameter λ, then
the sum T(X) = X 1 + ... + X n is a sufficient statistic for λ
Theorem
Let X 1 , ...., X n be a random sample with pdf f(x|θ). If
k
∑ i = 1
f ( x | θ ) = h(
x)
c(
θ )exp{ w ( θ ) t ( x)}
I
, where h( x)
≥ 0, c(
θ ) ≥ 0 then T ( X ) = ( t ( X ), L , t ( ))
is a sufficient
statistics for θ.
Example p376 ex2.14
i
i
A
( x)
∑
n
∑
n
i=
1 1 i i=
1 k
X
i
T(X) = (X (1) , ...., X (n) ) is better than T(X) = (X 1 , ...., X n ).
Minimal sufficiency
A sufficient statistic is minimal sufficient if it can be represented as a function of any
other sufficient statistic. In other words, S(X) is minimal sufficient if and only if
1. S(X) is sufficient, and
2. if T(X) is sufficient, then there exists a function f such that S(X) =
f(T(X)).
Intuitively, a minimal sufficient statistic most efficiently captures as much
information as is possible about the parameter θ.
A useful characterization of minimal sufficiency:
Theorem
Suppose that the density f θ exists. T(X) is minimal sufficient if and only if

For any sample point x,y f ( x | θ ) f ( y | θ ) is
independent of θ if and only if T(x) = T(y)
This follows as a direct consequence from the Fisher's factorization theorem stated
above.
Ex p377 ex2.16 ex2.17
A complete statistic is necessarily minimal sufficient.
Rao-Blackwell theorem
Sufficiency finds a useful application in the Rao-Blackwell theorem. It states that if
g(X) is any kind of estimator of θ, then typically the conditional expectation of g(X)
given T(X) is a better estimator of θ, and is never worse. Sometimes one can very
easily construct a very crude estimator g(X), and then evaluate that conditional
expected value to get an estimator that is in various senses optimal.