Foundations of Data Science

The above theorem tells us that the maximum of x T Ax subject to the constraint that
|x| 2 = 1 is λ 1 . Consider the problem of maximizing x T Ax subject to the additional restriction
that x is orthogonal to the first eigenvector. This is equivalent to maximizing
y t P t AP y subject to y being orthogonal to (1,0,. . . ,0), i.e. the first component of y being
0. This maximum is clearly λ 2 and occurs for y = (0, 1, 0, . . . , 0). The corresponding x is
the second column of P or the second eigenvector of A.
Similarly the maximum of x T Ax for p 1 T x = p 2 T x = · · · p s T x = 0 is λ s+1 and is
obtained for x = p s+1 .
12.7.4 Eigenvalues of the Sum of Two Symmetric Matrices
The min max theorem is useful in proving many other results. The following theorem
shows how adding a matrix B to a matrix A changes the eigenvalues of A. The theorem
is useful for determining the effect of a small perturbation on the eigenvalues of A.
Theorem 12.13 Let A and B be n × n symmetric matrices. Let C=A+B. Let α i , β i ,
and γ i denote the eigenvalues of A, B, and C respectively, where α 1 ≥ α 2 ≥ . . . α n and
similarly for β i , γ i . Then α s + β 1 ≥ γ s ≥ α s + β n .
Proof: By the min max theorem we have
α s =
min
max
r 1 ,...,r s−1
x
r i ⊥x
(x T Ax).
Suppose r 1 , r 2 , . . . , r s−1 attain the minimum in the expression. Then using the min max
theorem on C,
(
γ s ≤ max x T (A + B)x )
x⊥r 1 ,r 2 ,...r s−1
Therefore, γ s ≤ α s + β 1 .
≤
max (x T Ax) + max (x T Bx)
x⊥r 1 ,r 2 ,...r s−1
x⊥r 1 ,r 2 ,...r s−1
≤ α s + max
x (xT Bx) ≤ α s + β 1 .
An application of the result to A = C + (−B), gives α s ≤ γ s − β n . The eigenvalues
of -B are minus the eigenvalues of B and thus −β n is the largest eigenvalue. Hence
γ s ≥ α s + β n and combining inequalities yields α s + β 1 ≥ γ s ≥ α s + β n .
Lemma 12.14 Let A and B be n × n symmetric matrices. Let C=A+B. Let α i , β i ,
and γ i denote the eigenvalues of A, B, and C respectively, where α 1 ≥ α 2 ≥ . . . α n and
similarly for β i , γ i . Then γ r+s−1 ≤ α r + β s .
411