View File - University of Engineering and Technology, Taxila

3. For two independent sources, with states x r and x s , the probability ofx r and x s equals the product of their individual state’s probability,pðx r Þpðx s Þ. Consequently, the information conveyed by the jointevent of the two symbols is1I rs ¼ log apðx r Þpðx s Þ1¼ log apðx r Þ þ log 1apðx s Þ¼ I r þ I sð3:61ÞThis expression shows that the information conveyed by the joint event of thetwo symbols equals the sum of the information conveyed by each symbol.Although important, it is not the information associated with sourcestates but rather the average information associated with the source. Hence, theentropy of the DMC source can be written asHðX Þ¼ PMi¼1pðx i Þ log a1pðx i Þð3:62ÞExample 3.4: As an illustration of the meaning of ‘‘entropy,’’ consider abinary source that generates two symbols 0 and 1, with correspondingprobabilities b and ð1 bÞ. From (3.62), the binary source entropy isexpressed as1HðbÞ ¼b log 2b þð1 bÞ log 121 bð3:63ÞUsing this expression, we can draw the entropy function as in Fig. 3.20.Figure 3.20 shows that at the two extremes (i.e., when b ¼ 0 and 1) thesource output is always certain, meaning the entropy is zero; HðbÞ ¼0.However, if the symbols’ probabilities are equally probable—that is, whenb ¼ 0:5—the source output is maximally uncertain and the entropy ismaximum, HðbÞ ¼1. This condition demonstrates that the entropy can beinterpreted as a measure of uncertainty.It is easy to generalize for a DMC with a binary symmetric channel(BSC) in which all M symbols are equally likely to be transmitted and wouldhave an entropyHðX Þ¼log 2 ðMÞð3:64ÞCopyright © 2002 by Marcel Dekker, Inc. All Rights Reserved.