Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.266 18 — Crosswords and Codebreakingfor the rest of the transmission. The resulting correlations between the outputsof such pairs of machines provided a dribble of information-content fromwhich Turing and his co-workers extracted their daily 129 decibans.How to detect that two messages came from machines with a commonstate sequenceThe hypotheses are the null hypothesis, H 0 , which states that the machinesare in different states, and that the two plain messages are unrelated; and the‘match’ hypothesis, H 1 , which says that the machines are in the same state,and that the two plain messages are unrelated. No attempt is being madehere to infer what the state of either machine is. The data provided are thetwo cyphertexts x and y; let’s assume they both have length T and that thealphabet size is A (26 in Enigma). What is the probability of the data, giventhe two hypotheses?First, the null hypothesis. This hypothesis asserts that the two cyphertextsare given byx = x 1 x 2 x 3 . . . = c 1 (u 1 )c 2 (u 2 )c 3 (u 3 ) . . . (18.17)andy = y 1 y 2 y 3 . . . = c ′ 1(v 1 )c ′ 2(v 2 )c ′ 3(v 3 ) . . . , (18.18)where the codes c t and c ′ t are two unrelated time-varying permutations of thealphabet, and u 1 u 2 u 3 . . . and v 1 v 2 v 3 . . . are the plaintext messages. An exactcomputation of the probability of the data (x, y) would depend on a languagemodel of the plain text, and a model of the Enigma machine’s guts, but if weassume that each Enigma machine is an ideal random time-varying permutation,then the probability distribution of the two cyphertexts is uniform. Allcyphertexts are equally likely.( ) 1 2TP (x, y | H 0 ) = for all x, y of length T . (18.19)AWhat about H 1 ? This hypothesis asserts that a single time-varying permutationc t underlies bothandx = x 1 x 2 x 3 . . . = c 1 (u 1 )c 2 (u 2 )c 3 (u 3 ) . . . (18.20)y = y 1 y 2 y 3 . . . = c 1 (v 1 )c 2 (v 2 )c 3 (v 3 ) . . . . (18.21)What is the probability of the data (x, y)? We have to make some assumptionsabout the plaintext language. If it were the case that the plaintext languagewas completely random, then the probability of u 1 u 2 u 3 . . . and v 1 v 2 v 3 . . . wouldbe uniform, and so would that of x and y, so the probability P (x, y | H 1 )would be equal to P (x, y | H 0 ), and the two hypotheses H 0 and H 1 would beindistinguishable.We make progress by assuming that the plaintext is not completely random.Both plaintexts are written in a language, and that language has redundancies.Assume for example that particular plaintext letters are used moreoften than others. So, even though the two plaintext messages are unrelated,they are slightly more likely to use the same letters as each other; if H 1 is true,two synchronized letters from the two cyphertexts are slightly more likely tobe identical. Similarly, if a language uses particular bigrams and trigramsfrequently, then the two plaintext messages will occasionally contain the samebigrams and trigrams at the same time as each other, giving rise, if H 1 is true,