ICT12 3. Channel coding

ICT123. Channel coding25.9.2012 SXD3.1. Introduction3.2. Two decoding criteria3.3. General block codes3.4. Minimum distance decoding3.5. Computations

23.1. IntroductionSourceHelloworld!Sourceencoderu···1001···FeedbackchannelfChannelencoderx···1001011···ModulatorBinarydigitalChannelcodingis requiredto protectinformationNoiseReceiverHelloworld!Sourcedecoder···1001···x′Channeldecoderg···1000011···ychannelDemodulatorNoisy communications channel. I(S, R) = H(S) − H(S|R)againstthe effectsof noisein thechannel.

3For definiteness, let us assume that the source encoding alphabet andthe channel alphabet are both B = {0,1}. Later (Section 3.3) we will considera more general setup.Encoding. The channel encoder breaks the bit stream coming from thesource encoder into blocks u of k bits. Then, for each such informationblock, it produces a block x = f(u) of n > k bits, which is sent throughthe channel (f: B k → B n is assumed 1-to-1). The difference r = n − k iscalled the redundancy of the channel and the fact that x has r more bitsthan u is what will be used as protection against the channel errors.We will use C to denote the set of codewords and we will say that it is acode of type [n, k]. The rate of the code is defined as k/n.Since there are 2 k possible information blocks u, we also have 2 k codewordsx of length n (so |C| = 2 k ). Thus the proportion of codewords (orcode density) with respect to binary words of length n is 2 k /2 n =1/2 n−k = 1/2 r .

5Remark. If the received block y ∉ C, then the decoder knows that therehave been errors. In this case, if a feedback channel from the decoder tothe encoder is available, then there is a simple-minded error-correction:the decoder can ask the retransmission of x (this is called Automatic-Repeat-Request, of ARQ) until no errors are detected.Henceforth our interest will be mainly in encoders-decoders (codecs) thatdo not rely on ARQ, which in most cases is unfeasible. Such decoders arecapable of finding which bits of y are in error and then the correction isjust inverting those bits. So the main task will be in finding the positionsin which the errors occurred.As an example, we refer to the Hamming code [7,4,3] and the decoderstudied in T0.

65.2. Two decoding criteriaFirst some notations:P(x) be the probability of sending x ∈ C (channel input probabilities)P(y|x) probability of receiving y ∈ B n when x ∈ C is sent.P(x′|y) probability that x′ ∈ C was sent, given that y is received.P(y) = 1/2 n (We regard the elements of B n as equiprobable).nNote that for a BSC, P(y|x) = ∏j=1P(y j |x j ), with P(1|0) = P(0|1) = pand P(0|0) = P(1|1) = 1 − p.Minimum-error criterion (ME). According to this criterion, the task of thedecoder is to find x ′ ∈ C that maximizes, given a received word y,P(x ′ |y).Maximum likelihood (ML). Under this criterion, the task is to find x ′ ∈ Cthat maximizes P(y|x ′ ).

7Proposition. The two decoding criteria (ME and ML) are equivalent if thechannel inputs are equiprobable.Proof. By Bayes’ rule we haveP(x′|y) = P(yx′ )P(x ′ )P(y)= 2 n P(y|x ′ )P(x ′ ). [∗]It follows that the ME criterion is equivalent to maximizeP(y|x ′ )P(x ′ ) for x ′ ∈ C.If the x′ are equiprobable (so P(x ′ ) = 1/2 k ), then it is clear that thismaximization coincides with the maximization of the ML criterion.Remark. The maximization of P(y|x ′ )P(x ′ ) for x ′ ∈ C (ME criterion) requiresknowledge of the channel input probabilities P(x ′ ) and of thechannel probabilities (to find P(y|x ′ )), whereas the ML criterion onlyneeds the channel probabilities P(y|x ′ ).Henceforth we will assume that P(x ′ ) = 1/2 k and hence the two criteriaare equivalent. After source coding this is a reasonable hypothesis.

83.3. General block codesThe Hamming spaceIf T is a finite alphabet (T from transmission) and n is a positive integer,the elements of the set T n are the words of length n formed with the elementsof T. Often the elements of T n are also called vectors of length n.The Hamming distance, hd(x, y), between two words is the number ofindices i in the range 1. . n with x i ≠ y i .The Hamming distance is a distance in the set T n , which means that itsatisfies the following properties:1. hd(x, y) = hd(y, x) for all x, y ∈ T n .2. hd(x, y) > 0 if x ≠ y and hd(x, y) = 0 if x = y.3. hd(x, z) ≤ hd(x, y) + hd(y, z) for all x, y, z ∈ T n .The properties 1 and 2 are clear from the definitions.E.3.1. Prove the property 3.

9CodesA (block) T-code of length n is a subset C of T n . If M is the cardinal of C,we say that C is of type (n, M).We will set |T| = q (cardinal of the transmission alphabet) and we saythat T has arity q, or that T is a q-ary alphabet (binary and ternary whenq = 2 and q = 3, respectively).If we want to indicate q in the type of the code, we will write (n, M) q . Forbinary codes, q is often omitted.The minimum distance of C, d = d C , is the minimum of the valueshd(x, x′) for all pairs x, x ′ ∈ C such that x ≠ x′.In block error-correcting, the stream of symbols output by the compressoris broken into blocks of length k. These blocks are called informationvectors or messages.If the symbols belong to the alphabet T, then the set of messages isM = T k .

10CodersA coder of type [n, k] is a one-to-one map f: T k → T n . The image of thismap is a subset C of T n which will be called the code of f.If d = d C , we say that C has type [n, k, d], and often we will writeC ~ [n, k, d]to denote this (or [n, k, d] q if we want to indicate arity of T).Alternatively, we may say that the type is n, q k , d q, as the cardinal of C,which is equal to the cardinal of T k , is q k .For example, in the case of the repetition code of order 3 we have k = 1,M = {0,1}, n = 3 and C = {000,111}. Therefore C ∼ [3, 1, 3].Similarly, for the Hamming encoder f: B 4 → B 7 we have k = 4, n = 7and d = 3, and so the corresponding code has type [7,4,3].

13It is clear, directly from the definitions, that if hd(x, y) ≤ t (this conditionmeans that the number of errors produced by the channel is not greaterthan t), then x´ = x. Consequently, the coding/decoding scheme guaranteesthat any error pattern with at most t errors is corrected.Note that if the number of errors is > t, then it can happen that y is decodablebut with x ′ ≠ x. In this case we speak of an undetectable error.For example, if 000 is the vector sent in the binary repetition code andthe received vector is 011, then the decoded symbol is 1, which does notagree with the sent information symbol 0.

143.4. Minimum distance decodingNotation. Given w ∈ T n and a non-negative integerr, we setB(w, r) = {z ∈ T n |hd(w, z) ≤ r}.The set B(w, r) is called the (Hamming) ball of center w and radius r.Examples. B(000,0) = {000}, B(000,1) = {000, 100, 010, 001},B(000,2) = {000,100, 010, 001,110, 101, 110}.B(1111111,1) = {1111111,0111111,1011111, 1101111,1110111,1111011,1111101,1111110}Lemma. |B(w, r)| = vol q (n, r), where vol q (n, r) = ∑ n j=0 j (q − 1) j .For q = 2, vol 2 (n, r) = ∑ n j=0 j .E.3.2. Write a proof of the lemma.rrwrB(w, r)

15If C = {x 1 , … , x M }, let D i = B(x i , t), where t = ⌊(d − 1)/2⌋, with d theminimum distance of C. So t is the greatest integer such that 2t ≤ d − 1,or 2t < d.d 3 4 5 6 7t 1 1 2 2 3Since hdx j , x k ≥ d for j ≠ k, it follows that C ∩ D i = {x i }.It is also true that D j ∩ D k = ∅ if j ≠ k. Indeed, if y ∈ D j ∩ D k , thenhdx j , x k ≤ hdx j , y + hd(y, x k ) ≤ t + t < d.But by definition of d this can only occur if x j = x k .

16D 6D 5x 6D 1D 7x 1x 7D 2x 2D 3Therefore, if we set D C = ∐there is a unique mapg: D C → Csuch that g(y) = x i for all y ∈ D i .xi ∈C D i ,x 5x 4D 4x 3We extend g to T n by setting g(y) =?if y ∉ D C .By construction, g is a decoder of C and it corrects t errors. This decoderg is called the minimum distance decoder of C.To summarize:• g(y) is the word x′ ∈ C such that hd(y, x ′ ) ≤ t, if such an x′ exists,and otherwise y is non-decodable for g.• If y is decodable and g(y) = x′, then hd(x, y) > t for all x ∈ C − {x′}.

17Remark. The usefulness of the minimum distance decoder arises fromthe fact that we can hope, in most ordinary situations, that the transmissionsx ↦ y that lead to a decoder error (y ∉ D), or to undetectable errors(y ∈ D, but hd(y, x) > t) will in general be much less likely than thetransmissions x ↦ y for which y is decodable and g(y) = x.The minimum distance decoder maximizes the likelihood of correcting errorsif all the transmission symbols have the same probability of being alteredby the channel noise and if the q − 1 possible errors for a givensymbol are equally likely. If these conditions are satisfied, the channel issaid to be a (q-ary) symmetric channel.Unless otherwise declared, henceforth we will understand that ‘channel’means ‘symmetric channel’.

18Remark. From the computational point of view, the minimum distancedecoder, as defined above, is inefficient in general, even if d C is known,for it has to calculate hd(y, x), for x ∈ C, until hd(y, x) ≤ t, so that theaverage number of distances that have to be calculated is of the order of|C| = q k .Note also that this requires having generated the q k elements of C.But we also have to say that the progress in block coding theory in thelast sixty years can be seen, to a considerable extent, as a series of milestonesthat signal conceptual and algorithmic improvements that makepossible to deal with the minimum distance decoder, for large classes ofcodes, in ever more efficient ways.