Entropy and Mutual Information

Convexity and Concavity of MII(X; Y ) is a concave function of p(x) given p(y|x)and a convex function of p(y|x) given p(x).To prove I(X; Y ) is concave in p(x) given p(y|x),we use I(X; Y ) = H(Y ) − H(Y |X).H(Y ){λp 1 (x) + (1 − λ)p 2 (x)} = H(Y ){λp 1 (y) + (1 − λ)p 2 (y)}≥ λH(Y ){p 1 (y)} + (1 − λ)H(Y ){p 2 (y)}= λH(Y ){p 1 (x)} + (1 − λ)H(Y ){p 2 (x)},So the first term H(Y ) is concave in p(x). Thesecond term H(Y |X) is linear in p(x), which is bothconcave and convex. Therefore the rhs is a concavefunctions of p(x).Entropy and Mutual Information – p. 24