Entropy and Mutual Information

More documents

Recommendations

Info

Independence Bound on EntropyLet X 1:n be n random variables, thenH(X 1:n ) ≤n∑i=1H(X i ).To prove, expand H(X 1:n ) as the sum of conditionalentropies and then bound the conditional entropy byunconditional entropy.Entropy and Mutual Information – p. 20
Log Sum InequalityLet a 1:n , b 1:n be non-negative numbers, thenn∑i=1a i log a ib i≥ (n∑i=1a i ) log∑ ni=1 a i∑ ni=1 b i,with equality if and only if a ib iis constant.To prove, first note that the function f(x) = x log xis strictly convex for positive x. Define α i =b i ∑j b jand x i = a ib i. From ∑ α i f(x i ) ≥ f( ∑ α i x i ), the logsum inequality follows.Entropy and Mutual Information – p. 21
Page 1: Entropy and Mutual InformationNotes
Page 5 and 6: Mutual InformationThe mutual inform
Page 7 and 8: Conditional Mutual InformationThe c
Page 9 and 10: Chain Rules of Mutual InformationIt
Page 11 and 12: Relative EntropyWe have the chain r
Page 13 and 14: Relationships between H and IFor ra
Page 15 and 16: Sufficient Condition for ConvexityI
Page 17 and 18: Proof of Jensen’s InequalityWe pr
Page 19: Bound on EntropyLet X be the range
Page 23 and 24: Concavity of EntropyH(X) is a conca
Page 25 and 26: Convexity and Concavity of MITo pro
Page 27 and 28: Fano’s InequalityLet ˆX = g(Y )

Entropy and Mutual Information

Create successful ePaper yourself

Delete template?

Save as template?