Foundations of Data Science

Theorem 12.11 (The fundamental theorem of symmetric matrices) A real matrix
A is orthogonally diagonalizable if and only if A is symmetric.
Proof: (if) Assume A is orthogonally diagonalizable.
D = P −1 AP . Since P −1 = P T , we get
Then there exists P such that
A = P DP −1 = P DP T
which implies
and hence A is symmetric.
(only if) Already roved.
A T = (P DP T ) T = P DP T = A
Note that a nonsymmetric matrix may not be diagonalizable, it may have eigenvalues
that are not real, and the number of linearly independent eigenvectors corresponding to
an eigenvalue may be less than its multiplicity. For example, the matrix
⎛
⎝
1 1 0
0 1 1
1 0 1
( )
has eigenvalues 2, 1 + i √ 3
, and 1 − i √ 3
1 2
. The matrix has characteristic equation
2 2 2 2 0 1
(1 − λ) 2 = 0 and thus has eigenvalue 1 with multiplicity 2 but has only ( one ) linearly
1
independent eigenvector associated with the eigenvalue 1, namely x = c c ≠ 0.
0
Neither of these situations is possible for a symmetric matrix.
12.7.2 Relationship between SVD and Eigen Decomposition
The singular value decomposition exists for any n × d matrix whereas the eigenvalue
decomposition exists only for certain square matrices. For symmetric matrices the decompositions
are essentially the same.
The singular values of a matrix are always positive since they are the sum of squares
of the projection of a row of a matrix onto a singular vector. Given a symmetric matrix,
the eigenvalues can be positive or negative. If A is a symmetric matrix with eigenvalue
decomposition A = V E D E VE
T and singular value decomposition A = U SD S VS T , what is
the relationship between D E and D S , and between V E and V S , and between U S and V E ?
Observe that if A can be expressed as QDQ T where Q is orthonormal and D is diagonal,
then AQ = QD. That is, each column of Q is an eigenvector and the elements of D
are the eigenvalues. Thus, if the eigenvalues of A are distinct, then Q is unique up to
a permutation of columns. If an eigenvalue has multiplicity k, then the space spanned
the k columns is unique. In the following we will use the term essentially unique to
⎞
⎠
408