Foundations of Data Science

capture this situation. Now AA T = U S DS 2 U S T and AT A = V S DS 2 V S T . By an argument
similar to the one above, U S and V S are essentially unique and are the eigenvectors or
negatives of the eigenvectors of A and A T . The eigenvalues of AA T or A T A are the squares
of the eigenvalues of A. If A is not positive semi definite and has negative eigenvalues,
then in the singular value decomposition A = U S D S V S , some of the left singular vectors
are the negatives of the eigenvectors. Let S be a diagonal matrix with ±1 ′ s on the
diagonal depending on whether the corresponding eigenvalue is positive or negative. Then
A = (U S S)(SD S )V S where U S S = V E and SD S = D E .
12.7.3 Extremal Properties of Eigenvalues
In this section we derive a min max characterization of eigenvalues that implies that
the largest eigenvalue of a symmetric matrix A has a value equal to the maximum of
x T Ax over all vectors x of unit length. That is, the largest eigenvalue of A equals the
2-norm of A. If A is a real symmetric matrix there exists an orthogonal matrix P that
diagonalizes A. Thus
P T AP = D
where D is a diagonal matrix with the eigenvalues of A, λ 1 ≥ λ 2 ≥ · · · ≥ λ n , on its
diagonal. Rather than working with A, it is easier to work with the diagonal matrix D.
This will be an important technique that will simplify many proofs.
Consider maximizing x T Ax subject to the conditions
1.
n∑
x 2 i = 1
i=1
2. r T i x = 0, 1 ≤ i ≤ s
where the r i are any set of nonzero vectors. We ask over all possible sets {r i |1 ≤ i ≤ s}
of s vectors, what is the minimum value assumed by this maximum.
Theorem 12.12 (Min max theorem) For a symmetric matrix A, min
max
r 1 ,...,r s
x
r i ⊥x
(x t Ax) =
λ s+1 where the minimum is over all sets {r 1 , r 2 , . . . , r s } of s nonzero vectors and the
maximum is over all unit vectors x orthogonal to the s nonzero vectors.
Proof: A is orthogonally diagonalizable. Let P satisfy P T P = I and P T AP = D, D
diagonal. Let y = P T x. Then x = P y and
x T Ax = y T P T AP y = y T Dy =
n∑
λ i yi
2
i=1
Since there is a one-to-one correspondence between unit vectors x and y, maximizing
x T Ax subject to ∑ ∑
x 2 i = 1 is equivalent to maximizing n λ i yi 2 subject to ∑ yi 2 = 1. Since
409
i=1