Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.608 C — Some Mathematics⎡⎢⎣⎡Matrix⎣ 1 2 01 1 00 0 1⎤⎦Eigenvalues and eigenvectors e L , e R⎡2.41 1 −0.41⎣ .58 ⎤ ⎡.82⎦⎣ .82 ⎤ ⎡.58⎦⎣ 0 ⎤ ⎡0⎦⎣ 0 ⎤ ⎡0⎦⎣ −.58 ⎤ ⎡.82⎦⎣ −.82 ⎤.58⎦0 0 1 1 0 0[ ] 0 1[ 1.62 ] [ ] [ −0.62 ] [ ].53 .53 .85 .851 1.85 .85 −.53 −.531 1 0 00 0 0 11 0 0 00 0 1 1⎤⎥⎦⎡ 1.62 ⎤ ⎡ ⎤ ⎡ 0.5+0.9i ⎤ ⎡ ⎤ ⎡ 0.5−0.9i ⎤ ⎡ ⎤ ⎡ −0.62 ⎤ ⎡ ⎤.60 .60 .1−.5i .1−.5i .1+.5i .1+.5i .37 .37⎢.37⎥ ⎢.37⎥ ⎢−.3−.4i⎥ ⎢ .3+.4i⎥ ⎢−.3+.4i⎥ ⎢ .3−.4i⎥ ⎢−.60⎥ ⎢−.60⎥⎣.37⎦⎣.37⎦⎣ .3+.4i⎦⎣−.3−.4i⎦⎣ .3−.4i⎦⎣−.3+.4i⎦⎣−.60⎦⎣−.60⎦.60 .60 −.1+.5i −.1+.5i −.1−.5i −.1−.5i .37 .37Table C.4. Some matrices and their eigenvectors.Matrix[0 .381 .62⎡⎣ 0 .35 0 ⎤0 0 .46 ⎦1 .65 .54Eigenvalues and eigenvectors e L , e R] [ ] 1 [ ] [ −0.38 ] [ ].71 .36 −.93 −.71.71 .93 .36 .71⎡ 1 −0.2−0.3i −0.2+0.3i⎣ .58 ⎤ ⎡.58⎦⎣ .14 ⎤ ⎡.41⎦⎣ −.8+.1i ⎤ ⎡−.2−.5i⎦⎣.2−.5i ⎤ ⎡−.6+.2i⎦⎣ −.8−.1i ⎤ ⎡−.2+.5i⎦⎣.2+.5i ⎤−.6−.2i⎦.58 .90 .2+.2i .4+.3i .2−.2i .4−.3iTable C.5. Transition probability matrices for generating random paths through trellises.This property can be rewritten in terms of the all-ones vector n = (1, 1, . . . , 1) T :n T Q = n T .(C.7)So n is the principal left-eigenvector of Q with eigenvalue λ 1 = 1.e (1)L= n. (C.8)Because it is a non-negative matrix, Q has a principal right-eigenvector thatis non-negative, e (1)R. Generically, for Markov processes that are ergodic, thiseigenvector is the only right-eigenvector with eigenvalue of magnitude 1 (seetable C.6 for illustrative exceptions). This vector, if we normalize it such thate (1)R ·n = 1, is called the invariant distribution of the transition probabilitymatrix. It is the probability density that is left unchanged under Q. Unlikethe principal left-eigenvector, which we explicitly identified above, we can’tusually identify the principal right-eigenvector without computation.The matrix may have up to N − 1 other right-eigenvectors all of which areorthogonal to the left-eigenvector n, that is, they are zero-sum vectors.C.3 Perturbation theoryPerturbation theory is not used in this book, but it is useful in this book’sfields. In this section we derive first-order perturbation theory for the eigenvectorsand eigenvalues of square, not necessarily symmetric, matrices. Mostpresentations of perturbation theory focus on symmetric matrices, but nonsymmetricmatrices (such as transition matrices) also deserve to be perturbed!