Scribe 9 - Classes

i.e. the density f can be factored into a product such that one factor, h, does not depend on θ
and the other factor, which does depend on θ, depends on X only through T(X).
Example 2: If X 1 , . . . , X n are independent and uniformly distributed on the interval [0, θ],
then T (X) = max(X 1 , . . . , X n ) is sufficient for θ the sample maximum is a sufficient statistic
for the population maximum.
To see this, consider the joint probability density function of X = (X 1 , . . . , X n ). Because the
observations are independent, the pdf can be written as a product of individual densities.
f X (x 1 , . . . , x n ) = 1 θ 1 {0≤x 1 ≤θ} · · · 1
θ 1 {0≤x n≤θ} (1)
= 1
θ n 1 {0≤min{x i }}1 {max{xi }≤θ} (2)
where 1 {...} is the indicator function. Thus the density takes form required by the FisherNeyman
factorization theorem, where h(x) = 1 {min{xi }≥0}, and the rest of the expression is a function
of only θ and T (x) = max{x i }.
FANO’S INEQUALITY
This scenario is viewed as a communication link between a source and a receiver. X is data
transmitted from the source, and Y is received data at the receiver. The channel is noisy. If we
estimate X from Y, what is P (X ≠ ˆX) ?
H(X|Y )−H(Pe)
log(N−1)
≥
H(X|Y )−1
log(N−1)
P e ≡ P (X ≠ ˆX) ≥
N is the size of the outcome set of X. This inequality shows a lower bound on error of
estimation of X from Y.
Proof: Let E = I(X ≠ hatX) is an indicator function, so E is a binary random variable as:
P (E = 1) = P e and P (E = 0) = 1 − P e and H(E) = H(P e )
We have: H(E, X|Y ) = H(X|Y ) + X(E|X, Y ) = H(X, Y ) (1)(because knowing X and Y, we
know E).
Also, we have: H(E, X|Y ) = H(E|Y ) + H(X|E, Y )
≤ H(E) + H(X|Y, E = 0) · (1 − P e ) + H(X|Y, E = 1) · P e
= H(P e ) + H(X|Y, E = 1) · P e (since H(X|Y, E = 0) = 0 .)
≤ H(P e ) + log(N − 1) · P e (2) (since H(X—Y,E=1) is maximized when X is equally likely in
other N-1 choices)
From (1) and (2), we have: H(X|Y ) ≤ H(P e ) + log(N − 1) · P e
H(X|Y )−H(Pe)
H(X|Y )−1
log(N−1)
⇒ P e ≥
log(N−1)
≥
The last inequality holds since H(P e ) ≤ 1
FANO’S INEQUALITY EXAMPLE
Consider a source X = 1 : 5, P (X) = [0.35, 0.35, 0.1, 0.1, 0.1] T , also let Y=1,2 if X ≤ 2 then
y=x with probability 6/7, while if x¿2 then y=1 or 2 with equal probability.
Our best strategy is to guess hat(x) = y. We now calculate tha actual error probability and
the bound given by Fano’s inequality.
2