 t - VTU e-Learning Centre

Define the capacity of noisy DMC
Defined as –
CAPACITY OF A DMC
Session – 10
The maximum possible rate of information transmission over the channel.
In equation form –
P(
x )
D
C Max -----(1)
t
i.e., maximised over a set of input probabilities P(x) for the discrete r . v. X
What is Dt?
Dt: Ave. rate of information transmission over the channel defined as
H( x)
H(
x / y)
r
bits / sec. -----(2)
Dt s
Eqn. (1) becomes
P(
x)
H ( x)
H(
x / y)
r
C Max
-----(3)
Illustrative Examples
1. For the channel shown determine C,
Input
X
Solution:
0
1
2
3
Step – 1: Calculate H(x) = Pi
log Pi
i 0
Substituting we get,
1
p
p
1
3
s
0
1
1 – p = q
(1 – p) = q
2
3
Output
Y
P (x = 0) = P
P (x = 1) = Q
P (x = 2) = Q
P (x = 3) = P
2P + 2Q = 1

H(
x)
[ P(
x
P(
x
0)
log P(
x
2)
log P(
x
0)
2)
P(
x
P(
x
1)
log P(
x
1)
3)
log P(
x
= - [P log Q + Q log Q + Q log Q + P log P]
2P log P 2Q
log Q
H( x)
2
2
bits/sym. ----(4)
Step – 2: Calculate H(x/y) = P ( xy)
log P ( x / y)
Note, i & j can take values
i 1 1 2 2
j 1 2 1 2
H(
x / y)
[ P(
x
P(
x
P(
x
2,
2,
y
i
1,
y 1)
log p(
x 1 / y 1)
P(
x 1,
y 2)
log p(
x 1 / y 2)
Step – 3: Calculate P(x/y) using,
(i) P(x = 1 / y = 1) =
Where,
Input
X
0
1
2
3
2)
log ( x
j
y 1)
log p(
x 2 / y 1)
P(
x
P(y=1) = P(x=1, y=1) + P(x=2, y=1)
2 / y 2)]
P ( x / y)
1)
. P(
y 1/
x 1)
P(
y 1)
3)]
P(
x)
. P(
y / x)
P(
y)
= P(x=1) . P(y=1 / x=1) + P(x=2,) . P(y=1 / x=2)
= Q . p + Q . q
= QP + Q (1 – p)
P(
y 1)
Q
1
p
p
1
0
1
1 – p = q
(1 – p) = q
2
3
Output
Y
. ----(5)

Q . p
P(
x 1 / y 1)
p
Q
Similarly calculate other p(x/y)’s
They are
(ii) P(x=1 / y=2) = q
(iii) P(x=2 / y=1) = q
(iv) P(x=2 / y=2) = p
Equation (5) becomes:
H(x/y) = [Q . p log p + Q . q log q + Q . q log q + Q . p log p]
= [Q . p log p + Q (1 - p) log q + Q (1 - p) . log q + Q . p log p]
= + [2Q . p log p + 2Q . q log q]
= + 2Q [p log p + q log q]
H(x/y) = 2Q . ----- (6)
Step – 4: By definition we have,
Dt = H(x) – H(x/y)
Substituting for H(x) & H(x/y) we get,
Dt = - 2P log2 P – 2Q . log2 Q – 2Q . ----- (7)
Recall, 2P + 2Q = 1, from which,
Q = ½ –P
1 1 1
D t - 2P log 2 P - 2
P
log
P
2
P
2 2 2
1
i.e., Dt - 2P log2
P (1-
2P) log P
2P
----- (8)
2
Step – 5: By definition of channel capacity we have,
C
Max
P(
x)
Q(
X)
D
t

d
(i) Find D t
dP
d d
1
i.e., D t
2P
log 2 P (
1
2P)
log
P
2P
dP dP
2
1
2
P
log2
e
2 . P log
2 e
2
1
( 2)
log
2 log P
2
2 P
P
1
2
P
2
Simplifying we get,
d
dP
D log P log Q
t
Setting this to zero, you get
log2 P = log2 Q +
OR
P = Q . 2 = Q . ,
Where, = 2
How to get the optimum values of P & Q?
2
Substitute, P = Q in 2P + 2Q = 1
i.e., 2Q + 2Q = 1
OR
1
Q =
2(
1 )
1
Hence, P = Q . = . =
2(
1 )
Step – 6: Channel capacity is,
2
2( 1 )
2P
log P 2Q
log Q 2
C Max
Q
P(
x)
Q(
X)
Substituting for P & Q we get,
-----(9)
-----(10)
-----(11)
-----(12)

C 2 . log
2(
1
)
2(
1
)
Simplifying you get,
1 1
. log
2(
1
)
2(
1
)
1
log 2
2(
1
)
C = log [2 (1+)] – log
or
2(
1
)
C = log2
Remember, = 2 and
= - (p log p + q log q)
What are the extreme values of p?
Case – I: Say p = 1
What does this case correspond to?
What is C for this case/
Note: = 0, = 2 0 = 1
2( 1
1)
C log2
log
1
2
[ 4]
If rs = 1/sec, C = 2 bits/sec.
Can this case be thought of in practice?
1
Case – II: p =
2
bits/sec. -----(13)
2 bits / sym.
1 1 1
Now, = – log log 1
2 2 2
= 2 = 2
2( 2 1)
log
log 2 [ 3]
1.
585 bits / sec.
2

Which are the symbols often used for the channel under discussion and why it is
so?
Input
X
OVER A NOISY CHANNEL ONE CAN NEVER SEND
INFORMATION WITHOUT ERRORS
2. For the discrete binary channel shown find H(x), H(y), H(x/y) & H(y/x) when
P(x=0) = 1/4, P(x=1) = 3/4 , = 0.75, & = 0.9.
Solution:
What type of channel in this?
H(x) = – i
p
t
i
log p
t
i
= – [P(x=0) log P(x=0) + P(x=1) log P(x=1)]
= – [p log p+ (1–p) log (1–p)]
Where, p = P(x=0) & (1-p) = p(x=1)
Substituting the values, H(x) = 0.8113 bits/sym.
H(y) = – i
0
1
2
3
0
X
(1 –)
(1 –)
Y
1
p
r
i
log p
r
i
= – [P(y=0) log P(y=0) + P(y=1) log P(y=1)]
r t
t
Recall, p 0 p 0 . p 00 p1
p10
1
p
p
1
0
1
1 – p = q
(1 – p) = q
2
3
0
1
Output
Y

P(y=0) = P(x=0) . P00 + P(x=1) P10
= p + (1 – p) (1 – )
Similarly, P(Y=1) = p (1 – ) + (1 – p)
H(Y) p ( 1
p)
( 1
)
log p ( 1
p)
( 1
)
p ( 1
)
( 1
)
log p ( 1
)
( 1
)
H(Y) = - Q1(p) . log Q1(p) – Q2(p) log Q2(p)
Substituting the values, H(y) = 0.82
H(y/x) = P ( x
Simplifying, you get,
i
j
i,
y j)
log P ( y j / x i)
H(y/x) = – [p . . log + (1 – p) (1 – ) log (1 – ) + p(1 – ) log (1 – )]
+ (1 – p) log ]
= – p [ . log – (1 – ) log (1 – ) – log – (1 – ) log (1 – )]
H(y/x) = PK + C
Compute ‘K’
+ terms not dependent on p
Rateat
whichinformnissupplied
Rateat
whichinformn
by
thechannel
to theReceiver
is
receivedby
theReceiver
i.e., H(x) – H(x/y) = H(y) – H(y/x)
Home Work:
Calculate Dt = H(y) – H(y/x)
d
Find Dt
dp
d
Set Dt to zero and obtain the condition on probability distbn.
dp
Finally compute ‘C’, the capacity of the channel.
Answer is: 344.25 b/second.

2. A channel model is shown
INPUT
X
1
0 0
p
What type of channel is this?
Write the channel matrix
P(Y/X)
Do you notice something special in this channel?
What is H(x) for this channel?
Say P(x=0) = P & P(x=1) = Q = (1 – P)
H(x) = – P log P – Q log Q = – P log P – (1 – P) log (1 – P)
What is H(y/x)?
H(y/x) = – [p log p + q log q]
What is H(y) for the channel?
H(y) = P(
y)
log P(
y)
= – [Pp log (Pp) + q log q + Q . p log (Q . p)]
What is H(x/y) for the channel?
H(y/x) = P(
xy)
log P(
x / y)
X
p
q
q
?
1
OUTPUT
Y
Y
0 ? 1
0 p q p
1 o q p

X
P(XY)
Y
y1 Y2 Y3
X1 Pp Pq O
X2 O Qq Qp
P(X/Y)
1 1
H(x/y) = + Pp log 1 + Pq log + Q . q log + Qp log 1
P
Q
X
Y
y1 Y2 Y3
X1 1 P O
X2 O Q 1