ECE 275B – Homework #2 – Due Tuesday 1/30/07 - UCSD DSP Lab

ECE 275B – Homework #2 – Due Tuesday 1/30/07
Reading
Kay Chapter 5; Moon Sections 10.5, 10.6, and 12.3.6. Make sure to understand every
example. The key ideas are a) Sufficient Statistics (especially as a form of data compression
and their importance for finding UMVUE’s); b) the Exponential Family of distributions
(especially the relationship to sufficient statistics); and c) the Rao–Blackwell–Lehmann–
Scheffe (RBLS) Theorem as a procedure for finding UMVUE’s. 1
Also read the excerpt from Steven Pinker’s book How the Mind Works (Norton, 1997) on
the distinction between the frequentist and subjective interpretations of probability, which
I have placed on the class website.
Problems
1. Let x = (x1, ··· ,xn) T be a vector of continuous iid samples. The order statistics
x ′ = (x ′ 1, ··· ,x ′ n) T = T (x) are defined to be the increasing rearrangement of the
elements of x,
x ′ 1 ≤ x ′ 2 ≤···≤x ′ n
for x ′ 1, ··· ,x ′ n an appropriate permutation of the elements of x. Because xi are assumed
to be independent realizations of a continuous random variable one can assume that
almost surely (i.e., with probability 1),
x ′ 1

A statistic g (not necessarily sufficient) is said to be necessary if the partition it induces
is coarser than the partition induced by any sufficient statistic. Equivalently, g is
necessary iff g is a measurable function of any particular sufficient statistic T , g = f(T ),
for some g and T dependent measurable function f.
With the above definitions of sufficiency and minimality, a minimal sufficient statistic
can then be defined to be a necessary and sufficient statistic.
Consider a statistic which induces a partition (equivalence class) on X via
x ∼ x ′
iff
pθ(x)
pθ(x ′ ) = g(x, x′ ) = function of x and x ′ independent of θ.
Prove that this statistic is necessary and sufficient and hence is a minimal sufficient
statistic. 3
3. Consider the pdf’s or pmf’s listed in problems 5.1, 5.2, 5.3, and 5.14 of Kay and in
10.6-12 of Moon. Place all of them into scalar exponential family form and for each
one give a sufficient statistic.
4. For a k–parameter exponential family distribution, show that if the elements of the
vector of sufficient statistics are affinely dependent (i.e., are not linearly independent
in the affine sense) then the order of the distribution can be reduced below k, showing
that the original vector of sufficient statistics is not minimal. Similarly, if the vector
of natural parameters as functions of the parameter vector θ are affinely dependent,
again show that the order of the distribution can be reduced. This is consistent with
the claim made in class that the k–dimensional vector of sufficient statistics for a kparameter
exponential family distribution is minimal iff the two affine independence
conditions are met.
5. Let y and x be jointly random vectors. For notational convenience, define the quantities
〈y〉 E {y} and y E {y| x}, and note that 〈y〉 = 〈y〉. 4 Prove that the covariance
matrices for y and y satisfy the relationship
Cov {y} =Cov{E {y| x}} +E{Cov {y| x}} =Cov{y} +E{Cov {y| x}} ,
which, in particular, implies the important fact that
Cov {y} ≥Cov {y} .
This fact shows that the conditional mean random variable y = E{y | x} is more
tightly distributed around its mean than y. Thus the processing of a measurement
x to form the conditional mean y will generally result in less uncertainty about an
3 This proof is essentially done in lecture, so this problem is basically an assignment in directed reading.
4 I.e., y is an unbiased estimator of y.
2

unknown random variable y, provided that x and y are not independent. Later, we
will see that the conditional mean provides an optimal reduction of the uncertainty in
y in the matrix mean-squared error sense.
6. Let the real scalar random variable y be uniformly distributed over the closed interval
[0, θ], for unknown θ, 0

Show that the minimal sufficient statistic derived in the previous problem can be
updated recursively as
T (yk) =T (yk−1)+bk y[k]
and give the functional form of bk. Also give the dimensions of T (yk) andbk, noting
that the minimal sufficient statistic T (yk) and the “gain” bk are constant in dimension
and do not increase in size as k increases.
4