Foundations of Data Science

is, the λ i are the eigenvalues of A and the p i are the corresponding eigenvectors. Since P
is invertible, the p i are linearly independent.
(if) Assume that A has n linearly independent eigenvectors p 1 , p 2 , . . . , p n with corresponding
eigenvalues λ 1 , λ 2 , . . . , λ n . Then Ap i = λ i p i and reversing the above steps
AP = [Ap 1 , Ap 2 , . . . , Ap n ] = [λ 1 p 1 , λ 2 p 2 , . . . λ n p n ] = P D.
Thus, AP = DP . Since the p i are linearly independent, P is invertible and hence A =
P −1 DP . Thus, A is diagonalizable.
It follows from the proof of the theorem that if A is diagonalizable and has eigenvalue
λ with multiplicity k, then there are k linearly independent eigenvectors associated with λ.
A matrix P is orthogonal if it is invertible and P −1 = P T . A matrix A is orthogonally
diagonalizable if there exists an orthogonal matrix P such that P −1 AP = D is diagonal.
If A is orthogonally diagonalizable, then A = P DP T and AP = P D. Thus, the columns
of P are the eigenvectors of A and the diagonal elements of D are the corresponding
eigenvalues.
If P is an orthogonal matrix, then P T AP and A are both representations of the same
linear transformation with respect to different bases. To see this, note that if e 1 , e 2 , . . . , e n
is the standard basis, then a ij is the component of Ae j along the direction e i , namely,
a ij = e i T Ae j . Thus, A defines a linear transformation by specifying the image under the
transformation of each basis vector. Denote by p j the j th column of P . It is easy to see that
(P T AP ) ij is the component of Ap j along the direction p i , namely, (P T AP ) ij = p i T Ap j .
Since P is orthogonal, the p j form a basis of the space and so P T AP represents the same
linear transformation as A, but in the basis p 1 , p 2 , . . . , p n .
Another remark is in order. Check that
A = P DP T =
n∑
d ii p i p T i .
i=1
Compare this with the singular value decomposition where
A =
n∑
σ i u i v T i ,
i=1
the only difference being that u i and v i can be different and indeed if A is not square,
they will certainly be.
12.7.1 Symmetric Matrices
For an arbitrary matrix, some of the eigenvalues may be complex. However, for a
symmetric matrix with real entries, all eigenvalues are real. The number of eigenvalues
406