Entropy and Mutual Information

More documents

Recommendations

Info

Convexity of Relative EntropyD(p||q) is convex in (p, q): For 0 ≤ λ ≤ 1 andprobability mass functions p 1 , p 2 , q 1 , q 2 ,D(λp 1 +(1−λ)p 2 ||λq 1 +(1−λ)q 2 ) ≤ λD(p 1 ||q 1 )+(1−λ)DTo prove, apply the log sum inequality to eachx ∈ X with a i = λ i p i (x) and b i = λ i q i (x), whereλ 1 = λ and λ 2 = 1 − λ. That is,( ∑i=1,2λ i p i (x)) log∑i=1,2 λ ip i (x)∑i=1,2 λ iq i (x) ≤ ∑i=1,2λ i p i (x) log λ ip i (x)λ i q i (x) .Summing over all x leads to the desired property.Entropy and Mutual Information – p. 22
Concavity of EntropyH(X) is a concave function of p(x).This follows from H(X) = log |X| − D(p(x)||u(x))and the convexity of the relative entropy. Here u(x)is the uniform distribution.Entropy and Mutual Information – p. 23
Page 1: Entropy and Mutual InformationNotes
Page 5 and 6: Mutual InformationThe mutual inform
Page 7 and 8: Conditional Mutual InformationThe c
Page 9 and 10: Chain Rules of Mutual InformationIt
Page 11 and 12: Relative EntropyWe have the chain r
Page 13 and 14: Relationships between H and IFor ra
Page 15 and 16: Sufficient Condition for ConvexityI
Page 17 and 18: Proof of Jensen’s InequalityWe pr
Page 19 and 20: Bound on EntropyLet X be the range
Page 21: Log Sum InequalityLet a 1:n , b 1:n
Page 25 and 26: Convexity and Concavity of MITo pro
Page 27 and 28: Fano’s InequalityLet ˆX = g(Y )

Entropy and Mutual Information

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?