Foundations of Data Science

Let Q be an orthonormal matrix.
Lemma 12.17 For all x, |Qx| = |x|.
Proof: |Qx| 2 2 = xT Q T Qx = x T x = |x| 2 2 .
Lemma 12.18 ||QA|| 2
= ||A|| 2
Proof: For all x, |Qx| = |x|. Replacing x by Ax, |QAx| = |Ax| and thus max
|x|=1 |QAx| =
max
|x|=1 |Ax|
Lemma 12.19 ||AB|| 2 F ≤ ||A||2 F ||B||2 F
Proof: Let a i be the i th column of A and let b j be the j th column of B. By the
Cauchy-Schwartz inequality ∥ ∥
∥a T i b j ≤ ‖a i ‖ ‖b j ‖. Thus ||AB|| 2 F = ∑ ∑ ∣ ∣
∣a T i b j 2
≤
i j
∑ ∑
‖a i ‖ 2 ‖b j ‖ 2 = ∑ ‖a i ‖ 2 ∑ ‖b j ‖ 2 = ||A|| 2 F ||B||2 F
i j
i
j
Lemma 12.20 ||QA|| F
= ||A|| F
Proof: ||QA|| 2 F = Tr(AT Q T QA) = Tr(A T A) = ||A|| 2 F .
Lemma 12.21 For real, symmetric matrix A with eigenvalues λ 1 ≥ λ 2 ≥ . . ., ‖A‖ 2 2 =
max(λ 2 1, λ 2 n) and ‖A‖ 2 F = λ2 1 + λ 2 2 + · · · + λ 2 n
Proof: Suppose the spectral decomposition of A is P DP T , where P is an orthogonal
matrix and D is diagonal. We saw that ||P T A|| 2 = ||A|| 2 . Applying this again,
||P T AP || 2 = ||A|| 2 . But, P T AP = D and clearly for a diagonal matrix D, ||D|| 2 is the
largest absolute value diagonal entry from which the first equation follows. The proof of
the second is analogous.
If A is real and symmetric and of rank k then ||A|| 2 2 ≤ ||A||2 F ≤ k ||A||2 2
Theorem 12.22 ||A|| 2 2 ≤ ||A||2 F ≤ k ||A||2 2
Proof: It is obvious for diagonal matrices that ||D|| 2 2 ≤ ||D||2 F ≤ k ||D||2 2 . Let D =
Q t AQ where Q is orthonormal. The result follows immediately since for Q orthonormal,
||QA|| 2
= ||A|| 2
and ||QA|| F
= ||A|| F
.
Real and symmetric are necessary for some of these theorems. This condition was
needed to express Σ = Q T AQ. For example, in Theorem 12.22 suppose A is the n × n
matrix
⎛ ⎞
1 1
1 1
A = ⎜ 0 ⎟
⎝ . . ⎠ .
1 1
||A|| 2
= 2 and ||A|| F
= √ 2n. But A is rank 2 and ||A|| F
> 2 ||A|| 2
for n > 8.
414