Entropy and Mutual Information

More documents

Recommendations

Info

ConvexityWe will derive some inequalities in informationtheory. We begin by the concept of convexity.A set is convex if every line segment between twopoints in the set is a subset of the set.A function f(x) is convex over an interval (a, b) iffor every x 1 , x 2 ∈ (a, b) and 0 ≤ λ ≤ 1, it is true thatf(λx 1 + (1 − λ)x 2 ) ≤ λf(x 1 ) + (1 − λ)f(x 2 ).f is strictly convex if the equality holds only whenλ = 0, 1.A function f(x) is concave if −f(x) is convex.Entropy and Mutual Information – p. 14
Sufficient Condition for ConvexityIf a function f has non-negative (positive) secondderivatives everywhere, then f is (strictly) convex.This can be shown by Taylor’s expansion of f(x)f(x) = f(x 0 ) + f ′ (x 0 )(x − x 0 ) + f ′′ (x ∗ )2(x − x 0 ) 2around the point x 0 = λx 1 + (1 − λ)x 2 and evaluateat the points x = x 1 , x 2 .Entropy and Mutual Information – p. 15
Page 1: Entropy and Mutual InformationNotes
Page 5 and 6: Mutual InformationThe mutual inform
Page 7 and 8: Conditional Mutual InformationThe c
Page 9 and 10: Chain Rules of Mutual InformationIt
Page 11 and 12: Relative EntropyWe have the chain r
Page 13: Relationships between H and IFor ra
Page 17 and 18: Proof of Jensen’s InequalityWe pr
Page 19 and 20: Bound on EntropyLet X be the range
Page 21 and 22: Log Sum InequalityLet a 1:n , b 1:n
Page 23 and 24: Concavity of EntropyH(X) is a conca
Page 25 and 26: Convexity and Concavity of MITo pro
Page 27 and 28: Fano’s InequalityLet ˆX = g(Y )

Entropy and Mutual Information

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?