Scribe 9 - Classes

WEAKLAW OF LARGE NUMBERS
Given i.i.d X i , let S n = 1 n · ∑n
i=1 X i, then
E[S n ] = E[X] = µ, V ar(S n ) = 1 n · V ar(X) = 1 n · σ
As n increases, the variance of S n decreases, i.e., values of S n become clustered around the
probability
mean: S n −→
µ; P (|S n − µ| ≥ ɛ) n→∞ −→ 0
PROOF OF WEAK LAW OF LARGE NUMBERS
• Chebyshev’s Inequality P (|x − µ| ≥ ɛ) ≤ σ2
ɛ 2
Proof σ 2 = E[(X−µ) 2 ] = ∑ ∀x P (x)(x − µ)2 > ∑ |x−µ|≥ɛ
P (x) · (x − µ) > ɛ2·∑|x−µ|≥ɛ
P (x)
⇒ P (|x − µ| ≥ ɛ) ≤ σ2
ɛ 2
• WLLN
We apply Chebyshev: P (|S n − µ| ≥ ɛ) ≤ σ2
n·ɛ 2
n→∞
−→ 0
TYPICAL SET
Let x n is the i.i.d sequence of X i for i = 1 to n. Definition of typical set:
T n ɛ = x n ∈ X n : | − n −1 · logp(x n ) − H(X)| < ɛ with H(X) = H(X i )
EXAMPLE OF TYPICAL SET
Consider Bernoulli distribution with p
4