Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
178 11 — Error-Correcting Codes and Real Channels
How could such a channel be used to communicate information? Consider
transmitting a set of N real numbers {xn} N n=1 in a signal of duration T made
up of a weighted combination of orthonormal basis functions φn(t),
N
x(t) = xnφn(t), (11.7)
n=1
where T
0 dt φn(t)φm(t) = δnm. The receiver can then compute the scalars:
yn ≡
T
0
dt φn(t)y(t) = xn +
T
dt φn(t)n(t) (11.8)
≡
0
xn + nn (11.9)
for n = 1 . . . N. If there were no noise, then yn would equal xn. The white
Gaussian noise n(t) adds scalar noise nn to the estimate yn. This noise is
Gaussian:
nn ∼ Normal(0, N0/2), (11.10)
where N0 is the spectral density introduced above. Thus a continuous channel
used in this way is equivalent to the Gaussian channel defined at equation
(11.5). The power constraint T
0 dt [x(t)]2 ≤ P T defines a constraint on
the signal amplitudes xn,
n
x 2 n ≤ P T ⇒ x 2 n ≤
P T
N
. (11.11)
Before returning to the Gaussian channel, we define the bandwidth (measured
in Hertz) of the continuous channel to be:
N max
W = , (11.12)
2T
where N max is the maximum number of orthonormal functions that can be
produced in an interval of length T . This definition can be motivated by
imagining creating a band-limited signal of duration T from orthonormal cosine
and sine curves of maximum frequency W . The number of orthonormal
functions is N max = 2W T . This definition relates to the Nyquist sampling
theorem: if the highest frequency present in a signal is W , then the signal
can be fully determined from its values at a series of discrete sample points
separated by the Nyquist interval ∆t = 1/2W seconds.
So the use of a real continuous channel with bandwidth W , noise spectral
density N0, and power P is equivalent to N/T = 2W uses per second of a
Gaussian channel with noise level σ2 = N0/2 and subject to the signal power
constraint x2 n ≤ P/2W.
Definition of Eb/N0
φ1(t)
φ2(t)
φ3(t)
x(t)
Figure 11.1. Three basis functions,
and a weighted combination of
them, x(t) = N
n=1 xnφn(t), with
x1 = 0.4, x2 = − 0.2, and x3 = 0.1.
Imagine that the Gaussian channel yn = xn + nn is used with an encoding
system to transmit binary source bits at a rate of R bits per channel use. How
can we compare two encoding systems that have different rates of communication
R and that use different powers x2 n? Transmitting at a large rate R is
good; using small power is good too.
It is conventional to measure the rate-compensated signal-to-noise ratio by
the ratio of the power per source bit Eb = x2 n /R to the noise spectral density
N0: Eb/N0 is dimensionless, but it is
usually reported in the units of
Eb/N0 = decibels; the value given is
10 log10 Eb/N0.
x2n 2σ2 . (11.13)
R
Eb/N0 is one of the measures used to compare coding schemes for Gaussian
channels.